Hi Bruno, all, i've brought your initial work online at the "dev" pytest pages: http://pytest.org/dev/plugins_index.html I think it's a good start, thanks! My comments and wishes (others may comment as well!): - i'd collapse NAME and VERSION columns to save space, i.e. "pytest-bdd-0.6.1". - we should try to collect repository locations. maybe parsing for github/bitbucket urls would yield most of them automatically? Maybe we need to add some manually. - what about adding download numbers? - as to code organisation: you can leave it as is for now or (maybe better) put all code and generated files into a dedicated directory. A very interesting bit will be the "2.4.2 compat" and "dev" compat determination. In fact, i think "devpi" should help with that although i guess i need to improve some things. You can give using it a try with these steps: pip install devpi-client devpi use http://devpi.net/root/pypi devpi test -e py27 pytest-pep8 The last bit should download the pypi-version of pytest-pep8, unpack it and run tox for the "py27" environment and then post back test results (as a json file) to the devpi.net server. You can also ask about plugin "test" status: devpi list pytest-pep8 I just found out that this doesn't work for "pytest-flakes" yet although that does have a tox.ini. need to look into what's going on. In principle, devpi-server (an instance of which you are using above), provides a lot of detailed test info as a json, see here for example: http://devpi.net/+tests/6199353734615fde47d1fbfef1ebc737/toxresult The test results are stored as per the package MD5 at the moment. The whole test/list/json machinery of devpi needs some more work to become really nice, but it should already help as currently implemented, maybe best by using "devpi list" and filing feature requests against it :) For testing "development" versions, it should be enough to upload e.g. a "pytest-2.4.3.dev1" to a private index and then re-run the script that invokes something like the above devpi commands for all of the plugins. Now, for the plugins that don't have a tox.ini we could supply one out of band. For this "devpi test" would grow an option like "--toxini=mytox.ini" and would use that instead of expecting one in the downloaded unpacked distribution file. This tox.ini would probably just have a testenv that does "py.test --version" and "py.test --help" to see if something broke by just installing the plugin. So much for now, holger
Hi Holgen, On Sat, Oct 12, 2013 at 10:12 AM, holger krekel <holger@merlinux.eu> wrote:
My comments and wishes (others may comment as well!):
- i'd collapse NAME and VERSION columns to save space, i.e. "pytest-bdd-0.6.1".
- what about adding download numbers?
- as to code organisation: you can leave it as is for now or (maybe better) put all code and generated files into a dedicated directory
No problem, will work on those today. :)
- we should try to collect repository locations. maybe parsing for github/bitbucket urls would yield most of them automatically? Maybe we need to add some manually.
May I ask why we would need the repository locations? I mean, to work on the compatibility feature we can work directly with packages in pypi or devpi... A very interesting bit will be the "2.4.2 compat" and "dev" compat
determination. In fact, i think "devpi" should help with that although i guess i need to improve some things. You can give using it a try with these steps:
pip install devpi-client devpi use http://devpi.net/root/pypi devpi test -e py27 pytest-pep8
The last bit should download the pypi-version of pytest-pep8, unpack it and run tox for the "py27" environment and then post back test results (as a json file) to the devpi.net server. You can also ask about plugin "test" status:
devpi list pytest-pep8
I just found out that this doesn't work for "pytest-flakes" yet although that does have a tox.ini. need to look into what's going on. In principle, devpi-server (an instance of which you are using above), provides a lot of detailed test info as a json, see here for example:
http://devpi.net/+tests/6199353734615fde47d1fbfef1ebc737/toxresult
The test results are stored as per the package MD5 at the moment. The whole test/list/json machinery of devpi needs some more work to become really nice, but it should already help as currently implemented, maybe best by using "devpi list" and filing feature requests against it :)
For testing "development" versions, it should be enough to upload e.g. a "pytest-2.4.3.dev1" to a private index and then re-run the script that invokes something like the above devpi commands for all of the plugins.
Now, for the plugins that don't have a tox.ini we could supply one out of band. For this "devpi test" would grow an option like "--toxini=mytox.ini" and would use that instead of expecting one in the downloaded unpacked distribution file. This tox.ini would probably just have a testenv that does "py.test --version" and "py.test --help" to see if something broke by just installing the plugin.
Interesting! Will give it a try today. Will keep you posted as things move along. Cheers, Bruno.
So much for now, holger
On Mon, Oct 14, 2013 at 14:21 -0300, Bruno Oliveira wrote:
Hi Holgen,
hi Brunon! :)
On Sat, Oct 12, 2013 at 10:12 AM, holger krekel <holger@merlinux.eu> wrote:
My comments and wishes (others may comment as well!):
- i'd collapse NAME and VERSION columns to save space, i.e. "pytest-bdd-0.6.1".
- what about adding download numbers?
- as to code organisation: you can leave it as is for now or (maybe better) put all code and generated files into a dedicated directory
No problem, will work on those today. :)
- we should try to collect repository locations. maybe parsing for github/bitbucket urls would yield most of them automatically? Maybe we need to add some manually.
May I ask why we would need the repository locations? I mean, to work on the compatibility feature we can work directly with packages in pypi or devpi...
purely for the person reading the page and wanting to look at the code or do a PR.
A very interesting bit will be the "2.4.2 compat" and "dev" compat
determination. In fact, i think "devpi" should help with that although i guess i need to improve some things. You can give using it a try with these steps:
pip install devpi-client devpi use http://devpi.net/root/pypi devpi test -e py27 pytest-pep8
The last bit should download the pypi-version of pytest-pep8, unpack it and run tox for the "py27" environment and then post back test results (as a json file) to the devpi.net server. You can also ask about plugin "test" status:
devpi list pytest-pep8
I just found out that this doesn't work for "pytest-flakes" yet although that does have a tox.ini. need to look into what's going on. In principle, devpi-server (an instance of which you are using above), provides a lot of detailed test info as a json, see here for example:
http://devpi.net/+tests/6199353734615fde47d1fbfef1ebc737/toxresult
The test results are stored as per the package MD5 at the moment. The whole test/list/json machinery of devpi needs some more work to become really nice, but it should already help as currently implemented, maybe best by using "devpi list" and filing feature requests against it :)
For testing "development" versions, it should be enough to upload e.g. a "pytest-2.4.3.dev1" to a private index and then re-run the script that invokes something like the above devpi commands for all of the plugins.
Now, for the plugins that don't have a tox.ini we could supply one out of band. For this "devpi test" would grow an option like "--toxini=mytox.ini" and would use that instead of expecting one in the downloaded unpacked distribution file. This tox.ini would probably just have a testenv that does "py.test --version" and "py.test --help" to see if something broke by just installing the plugin.
Interesting! Will give it a try today. Will keep you posted as things move along.
great, i am travelling to Pycon DE tomorrow morning but should be online from time to time. best, holger
Cheers, Bruno.
So much for now, holger
Hi Holger, On Mon, Oct 14, 2013 at 5:32 PM, holger krekel <holger@merlinux.eu> wrote:
On Mon, Oct 14, 2013 at 14:21 -0300, Bruno Oliveira wrote:
Hi Holgen,
hi Brunon! :)
Hehehe, oops.
On Sat, Oct 12, 2013 at 10:12 AM, holger krekel <holger@merlinux.eu> wrote:
- i'd collapse NAME and VERSION columns to save space, i.e. "pytest-bdd-0.6.1".
- what about adding download numbers?
- as to code organisation: you can leave it as is for now or (maybe better) put all code and generated files into a dedicated directory
Done, opened a PR for discussion and reviewing. :)
- we should try to collect repository locations. maybe parsing for github/bitbucket urls would yield most of them automatically? Maybe we need to add some manually.
May I ask why we would need the repository locations? I mean, to work on the compatibility feature we can work directly with packages in pypi or devpi...
purely for the person reading the page and wanting to look at the code or do a PR.
Oh I see! Well, parsing the urls seems problematic, because we also have to take in account the username ("hpk42/pytest" for example). I looked at the GitHub and BitBucket's rest APIs to see if we could use them for searching, but it seems only GitHub at the moment has a search API (and in beta stage at that). I think we will have to maintain the repository list manually. Package authors can help as well by simply opening PRs adding repositories for their plugins I guess.
A very interesting bit will be the "2.4.2 compat" and "dev" compat
<snip>
Some late night quick hacking and I have come up with this proof of concept: https://www.travis-ci.org/nicoddemus/pytest-plugs As it is right now it serves only as a prove of concept, of course. The idea is using travis to handle running things for us, driving it using a script that downloads and run tests using tox. We can then collect json information from that and POST those results back to devpi. Also we can use its build matrix to test against different pytest versions, for example. I guess we can implement in devpi a concept similar to travis build status imagens, which we could then use in the plugins_index page. Against python 2.6 the script is failing, I have yet to investigate. great, i am travelling to Pycon DE tomorrow morning but should be online
from time to time.
Nice tripe and good PyCon! Cheers, Bruno.
On Mon, Oct 14, 2013 at 23:52 -0300, Bruno Oliveira wrote:
- i'd collapse NAME and VERSION columns to save space, i.e. "pytest-bdd-0.6.1".
- what about adding download numbers?
- as to code organisation: you can leave it as is for now or (maybe better) put all code and generated files into a dedicated directory
Done, opened a PR for discussion and reviewing. :)
only got to it now, looks good so far!
- we should try to collect repository locations. maybe parsing for github/bitbucket urls would yield most of them automatically? Maybe we need to add some manually.
May I ask why we would need the repository locations? I mean, to work on the compatibility feature we can work directly with packages in pypi or devpi...
purely for the person reading the page and wanting to look at the code or do a PR.
Oh I see! Well, parsing the urls seems problematic, because we also have to take in account the username ("hpk42/pytest" for example). I looked at the GitHub and BitBucket's rest APIs to see if we could use them for searching, but it seems only GitHub at the moment has a search API (and in beta stage at that).
I think we will have to maintain the repository list manually. Package authors can help as well by simply opening PRs adding repositories for their plugins I guess.
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
A very interesting bit will be the "2.4.2 compat" and "dev" compat
<snip>
Some late night quick hacking and I have come up with this proof of concept:
https://www.travis-ci.org/nicoddemus/pytest-plugs
As it is right now it serves only as a prove of concept, of course. The idea is using travis to handle running things for us, driving it using a script that downloads and run tests using tox. We can then collect json information from that and POST those results back to devpi. Also we can use its build matrix to test against different pytest versions, for example.
Hum, interesting. Is this mainly to make use of travis-ci's build hosts? What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I guess we can implement in devpi a concept similar to travis build status imagens, which we could then use in the plugins_index page.
Against python 2.6 the script is failing, I have yet to investigate.
great, i am travelling to Pycon DE tomorrow morning but should be online
from time to time.
Nice tripe and good PyCon!
Thanks! Got back but caught a flue ... cheers, holger
A very interesting bit will be the "2.4.2 compat" and "dev" compat
<snip>
Some late night quick hacking and I have come up with this proof of concept:
https://www.travis-ci.org/nicoddemus/pytest-plugs
As it is right now it serves only as a prove of concept, of course. The idea is using travis to handle running things for us, driving it using a script that downloads and run tests using tox. We can then collect json information from that and POST those results back to devpi. Also we can use its build matrix to test against different pytest versions, for example.
Hum, interesting. Is this mainly to make use of travis-ci's build hosts? What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I'd say the biggest advantage is to not execute stuff on your own server. They have to deal with that anyway. Regards, Florian Schulze
On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu> wrote:
Done, opened a PR for discussion and reviewing. :)
only got to it now, looks good so far!
Bummer, I thought merging it would automatically resolve the issue I created (as in GitHub)... oh well. :)
I think we will have to maintain the repository list manually. Package
authors can help as well by simply opening PRs adding repositories for their plugins I guess.
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
That's a possibility, for sure.
A very interesting bit will be the "2.4.2 compat" and "dev" compat
<snip>
Some late night quick hacking and I have come up with this proof of concept:
https://www.travis-ci.org/nicoddemus/pytest-plugs
As it is right now it serves only as a prove of concept, of course. The idea is using travis to handle running things for us, driving it using a script that downloads and run tests using tox. We can then collect json information from that and POST those results back to devpi. Also we can use its build matrix to test against different pytest versions, for example.
Hum, interesting. Is this mainly to make use of travis-ci's build hosts?
As Florian said, mainly so you don't need to have a dedicated host to run those scripts yourself.
What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I didn't use devpi at that point because of my limited knowledge of devpi at the moment, but that is certainly the next step. I will work on that next, and also work on generating a default tox.ini file for plugins that don't have one as you suggested.
I guess we can implement in devpi a concept similar to travis build status imagens, which we could then use in the plugins_index page.
Against python 2.6 the script is failing, I have yet to investigate.
I found the reason for that (http vs https) and now the job runs against Python 2.6, 2.7, 3.2 and 3.2. Thanks! Got back but caught a flue ...
It happens, hope you get better soon. :) Cheers, Bruno. On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu> wrote:
- i'd collapse NAME and VERSION columns to save space, i.e. "pytest-bdd-0.6.1".
- what about adding download numbers?
- as to code organisation: you can leave it as is for now or (maybe better) put all code and generated files into a dedicated
On Mon, Oct 14, 2013 at 23:52 -0300, Bruno Oliveira wrote: directory
Done, opened a PR for discussion and reviewing. :)
only got to it now, looks good so far!
- we should try to collect repository locations. maybe parsing for github/bitbucket urls would yield most of them automatically? Maybe we need to add some manually.
May I ask why we would need the repository locations? I mean, to work on the compatibility feature we can work directly with packages in pypi or devpi...
purely for the person reading the page and wanting to look at the code or do a PR.
Oh I see! Well, parsing the urls seems problematic, because we also have to take in account the username ("hpk42/pytest" for example). I looked at the GitHub and BitBucket's rest APIs to see if we could use them for searching, but it seems only GitHub at the moment has a search API (and in beta stage at that).
I think we will have to maintain the repository list manually. Package authors can help as well by simply opening PRs adding repositories for their plugins I guess.
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
A very interesting bit will be the "2.4.2 compat" and "dev" compat
<snip>
Some late night quick hacking and I have come up with this proof of concept:
https://www.travis-ci.org/nicoddemus/pytest-plugs
As it is right now it serves only as a prove of concept, of course. The idea is using travis to handle running things for us, driving it using a script that downloads and run tests using tox. We can then collect json information from that and POST those results back to devpi. Also we can use its build matrix to test against different pytest versions, for example.
Hum, interesting. Is this mainly to make use of travis-ci's build hosts? What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I guess we can implement in devpi a concept similar to travis build status imagens, which we could then use in the plugins_index page.
Against python 2.6 the script is failing, I have yet to investigate.
great, i am travelling to Pycon DE tomorrow morning but should be online
from time to time.
Nice tripe and good PyCon!
Thanks! Got back but caught a flue ...
cheers, holger
Hi Holger, all, On Sun, Oct 20, 2013 at 11:06 AM, Bruno Oliveira <nicoddemus@gmail.com>wrote:
On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu> wrote:
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
That's a possibility, for sure.
What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I didn't use devpi at that point because of my limited knowledge of devpi at the moment, but that is certainly the next step.
I will work on that next, and also work on generating a default tox.ini file for plugins that don't have one as you suggested.
I have updated the script to create a tox.ini file that just uses "py.test --help" as a test command for those packages that don't have one. Now the script tests all pytest plugins in pypi, please take a look at the summary at the bottom of the page: https://www.travis-ci.org/nicoddemus/pytest-plugs/jobs/12959702 I guess it would be useful to identify which plugins are using just "py.test --help" as smoke test and which ones have real tests, since the former doesn't guarantee that the plugin actually works on that python/pytest combination. I looked into devpi to simplify the download/unpack/test process and stumbled in a minor roadblock: is there anyway to provide a tox.ini file for those plugins that don't have one? This is the only issue I see right now with using devpi for this particular application.
| > I guess we can implement in devpi a concept similar to travis build status
imagens, which we could then use in the plugins_index page.
Against python 2.6 the script is failing, I have yet to investigate.
So, what do you think of the idea of using devpi.net to host test results and add an endpoint to the API to provide test status images? Or should I create another app to do this? Cheers, Bruno.
I found the reason for that (http vs https) and now the job runs against Python 2.6, 2.7, 3.2 and 3.2.
Thanks! Got back but caught a flue ...
It happens, hope you get better soon. :)
Cheers, Bruno.
On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu> wrote:
- i'd collapse NAME and VERSION columns to save space, i.e. "pytest-bdd-0.6.1".
- what about adding download numbers?
- as to code organisation: you can leave it as is for now or
(maybe
better) put all code and generated files into a dedicated
On Mon, Oct 14, 2013 at 23:52 -0300, Bruno Oliveira wrote: directory
Done, opened a PR for discussion and reviewing. :)
only got to it now, looks good so far!
- we should try to collect repository locations. maybe parsing for github/bitbucket urls would yield most of them automatically? Maybe we need to add some manually.
May I ask why we would need the repository locations? I mean, to work on the compatibility feature we can work directly with packages in pypi or devpi...
purely for the person reading the page and wanting to look at the code or do a PR.
Oh I see! Well, parsing the urls seems problematic, because we also have to take in account the username ("hpk42/pytest" for example). I looked at the GitHub and BitBucket's rest APIs to see if we could use them for searching, but it seems only GitHub at the moment has a search API (and in beta stage at that).
I think we will have to maintain the repository list manually. Package authors can help as well by simply opening PRs adding repositories for their plugins I guess.
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
A very interesting bit will be the "2.4.2 compat" and "dev" compat
<snip>
Some late night quick hacking and I have come up with this proof of concept:
https://www.travis-ci.org/nicoddemus/pytest-plugs
As it is right now it serves only as a prove of concept, of course. The idea is using travis to handle running things for us, driving it using a script that downloads and run tests using tox. We can then collect json information from that and POST those results back to devpi. Also we can use its build matrix to test against different pytest versions, for example.
Hum, interesting. Is this mainly to make use of travis-ci's build hosts? What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I guess we can implement in devpi a concept similar to travis build status imagens, which we could then use in the plugins_index page.
Against python 2.6 the script is failing, I have yet to investigate.
great, i am travelling to Pycon DE tomorrow morning but should be online
from time to time.
Nice tripe and good PyCon!
Thanks! Got back but caught a flue ...
cheers, holger
Hi Bruno, On Wed, Oct 23, 2013 at 21:51 -0200, Bruno Oliveira wrote:
Hi Holger, all,
On Sun, Oct 20, 2013 at 11:06 AM, Bruno Oliveira <nicoddemus@gmail.com>wrote:
On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu> wrote:
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
That's a possibility, for sure.
What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I didn't use devpi at that point because of my limited knowledge of devpi at the moment, but that is certainly the next step.
I will work on that next, and also work on generating a default tox.ini file for plugins that don't have one as you suggested.
I have updated the script to create a tox.ini file that just uses "py.test --help" as a test command for those packages that don't have one. Now the script tests all pytest plugins in pypi, please take a look at the summary at the bottom of the page:
https://www.travis-ci.org/nicoddemus/pytest-plugs/jobs/12959702
looks good! I guess having a py27 and py33 row would be great to also determine py3 compatibility/installability.
I guess it would be useful to identify which plugins are using just "py.test --help" as smoke test and which ones have real tests, since the former doesn't guarantee that the plugin actually works on that python/pytest combination.
I looked into devpi to simplify the download/unpack/test process and stumbled in a minor roadblock: is there anyway to provide a tox.ini file for those plugins that don't have one? This is the only issue I see right now with using devpi for this particular application.
Now, but i am going to add a "devpi test -c path-to-ini" option today and plan to release devpi-1.2 real soon now. I'll get back here when it's implemented.
| > I guess we can implement in devpi a concept similar to travis build status
imagens, which we could then use in the plugins_index page.
Against python 2.6 the script is failing, I have yet to investigate.
So, what do you think of the idea of using devpi.net to host test results and add an endpoint to the API to provide test status images? Or should I create another app to do this?
What do you mean with "test status images"? In general i am open to extending devpi-server to provide more test-status info -- just didn't have a chance to think about it in detail yet. I guess we could take the plugin testing bits as a starting point. best, holger
Cheers, Bruno.
I found the reason for that (http vs https) and now the job runs against Python 2.6, 2.7, 3.2 and 3.2.
Thanks! Got back but caught a flue ...
It happens, hope you get better soon. :)
Cheers, Bruno.
On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu> wrote:
> > - i'd collapse NAME and VERSION columns to save space, i.e. > "pytest-bdd-0.6.1". > > - what about adding download numbers? > > - as to code organisation: you can leave it as is for now or (maybe > better) put all code and generated files into a dedicated
On Mon, Oct 14, 2013 at 23:52 -0300, Bruno Oliveira wrote: directory
Done, opened a PR for discussion and reviewing. :)
only got to it now, looks good so far!
> - we should try to collect repository locations. maybe parsing > for github/bitbucket urls would yield most of them automatically? > Maybe we need to add some manually. >
May I ask why we would need the repository locations? I mean, to work on the compatibility feature we can work directly with packages in pypi or devpi...
purely for the person reading the page and wanting to look at the code or do a PR.
Oh I see! Well, parsing the urls seems problematic, because we also have to take in account the username ("hpk42/pytest" for example). I looked at the GitHub and BitBucket's rest APIs to see if we could use them for searching, but it seems only GitHub at the moment has a search API (and in beta stage at that).
I think we will have to maintain the repository list manually. Package authors can help as well by simply opening PRs adding repositories for their plugins I guess.
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
A very interesting bit will be the "2.4.2 compat" and "dev" compat
<snip>
Some late night quick hacking and I have come up with this proof of concept:
https://www.travis-ci.org/nicoddemus/pytest-plugs
As it is right now it serves only as a prove of concept, of course. The idea is using travis to handle running things for us, driving it using a script that downloads and run tests using tox. We can then collect json information from that and POST those results back to devpi. Also we can use its build matrix to test against different pytest versions, for example.
Hum, interesting. Is this mainly to make use of travis-ci's build hosts? What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I guess we can implement in devpi a concept similar to travis build status imagens, which we could then use in the plugins_index page.
Against python 2.6 the script is failing, I have yet to investigate.
great, i am travelling to Pycon DE tomorrow morning but should be online
from time to time.
Nice tripe and good PyCon!
Thanks! Got back but caught a flue ...
cheers, holger
Hi Holger, On Thu, Oct 24, 2013 at 4:52 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Bruno,
Hi Holger, all,
<snip> I have updated the script to create a tox.ini file that just uses "py.test --help" as a test command for those packages that don't have one. Now the script tests all pytest plugins in pypi, please take a look at the
On Wed, Oct 23, 2013 at 21:51 -0200, Bruno Oliveira wrote: summary
at the bottom of the page:
https://www.travis-ci.org/nicoddemus/pytest-plugs/jobs/12959702
looks good! I guess having a py27 and py33 row would be great to also determine py3 compatibility/installability.
Those are already in place as different build jobs, please take a look: https://www.travis-ci.org/nicoddemus/pytest-plugs Currently we have py27 and py33 against pytest 2.3.5 and 2.4.2. Adding more python/pytest versions is easy, we just have to add them to the .travis file. :) https://github.com/nicoddemus/pytest-plugs/blob/master/.travis.yml
I guess it would be useful to identify which plugins are using just "py.test --help" as smoke test and which ones have real tests, since the former doesn't guarantee that the plugin actually works on that python/pytest combination.
I looked into devpi to simplify the download/unpack/test process and stumbled in a minor roadblock: is there anyway to provide a tox.ini file for those plugins that don't have one? This is the only issue I see right now with using devpi for this particular application.
Now, but i am going to add a "devpi test -c path-to-ini" option today and plan to release devpi-1.2 real soon now. I'll get back here when it's implemented.
Perfect, that would greatly simplify things. Let me know if there is anything I can do to help.
So, what do you think of the idea of using devpi.net to host test results
and add an endpoint to the API to provide test status images? Or should I create another app to do this?
What do you mean with "test status images"?
In general i am open to extending devpi-server to provide more test-status
info -- just didn't have a chance to think about it in detail yet. I guess we could take the plugin testing bits as a starting point.
I mean like those badge images served by travis that give the current build status, for example: https://secure.travis-ci.org/hpk42/pytest.png I think we can follow the same approach, generating a badge that gives the full compatibility status, like a single "py 2.6 2.7 | pytest 2.3.5 | compatible" image, or even multiple images. But I guess this is too specific to pytest... Cheers, Bruno.
best, holger
Cheers, Bruno.
I found the reason for that (http vs https) and now the job runs against Python 2.6, 2.7, 3.2 and 3.2.
Thanks! Got back but caught a flue ...
It happens, hope you get better soon. :)
Cheers, Bruno.
On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu> wrote:
> > > > - i'd collapse NAME and VERSION columns to save space, i.e. > > "pytest-bdd-0.6.1". > > > > - what about adding download numbers? > > > > - as to code organisation: you can leave it as is for now or (maybe > > better) put all code and generated files into a dedicated
On Mon, Oct 14, 2013 at 23:52 -0300, Bruno Oliveira wrote: directory
>
Done, opened a PR for discussion and reviewing. :)
only got to it now, looks good so far!
> > - we should try to collect repository locations. maybe parsing > > for github/bitbucket urls would yield most of them automatically? > > Maybe we need to add some manually. > > > > May I ask why we would need the repository locations? I mean, to work on > the compatibility feature we can work directly with packages in pypi or > devpi...
purely for the person reading the page and wanting to look at the code or do a PR.
Oh I see! Well, parsing the urls seems problematic, because we also have to take in account the username ("hpk42/pytest" for example). I looked at the GitHub and BitBucket's rest APIs to see if we could use them for searching, but it seems only GitHub at the moment has a search API (and in beta stage at that).
I think we will have to maintain the repository list manually. Package authors can help as well by simply opening PRs adding repositories for their plugins I guess.
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
A very interesting bit will be the "2.4.2 compat" and "dev" compat > <snip>
Some late night quick hacking and I have come up with this proof of concept:
https://www.travis-ci.org/nicoddemus/pytest-plugs
As it is right now it serves only as a prove of concept, of course. The idea is using travis to handle running things for us, driving it using a script that downloads and run tests using tox. We can then collect json information from that and POST those results back to devpi. Also we can use its build matrix to test against different pytest versions, for example.
Hum, interesting. Is this mainly to make use of travis-ci's build hosts? What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I guess we can implement in devpi a concept similar to travis build status imagens, which we could then use in the plugins_index page.
Against python 2.6 the script is failing, I have yet to investigate.
great, i am travelling to Pycon DE tomorrow morning but should be online
from time to time.
Nice tripe and good PyCon!
Thanks! Got back but caught a flue ...
cheers, holger
On Thu, Oct 24, 2013 at 05:08 -0200, Bruno Oliveira wrote:
Hi Holger,
On Thu, Oct 24, 2013 at 4:52 AM, holger krekel <holger@merlinux.eu> wrote:
Hi Bruno,
Hi Holger, all,
<snip> I have updated the script to create a tox.ini file that just uses "py.test --help" as a test command for those packages that don't have one. Now the script tests all pytest plugins in pypi, please take a look at the
On Wed, Oct 23, 2013 at 21:51 -0200, Bruno Oliveira wrote: summary
at the bottom of the page:
https://www.travis-ci.org/nicoddemus/pytest-plugs/jobs/12959702
looks good! I guess having a py27 and py33 row would be great to also determine py3 compatibility/installability.
Those are already in place as different build jobs, please take a look:
Ah, see it now.
Currently we have py27 and py33 against pytest 2.3.5 and 2.4.2. Adding more python/pytest versions is easy, we just have to add them to the .travis file. :)
https://github.com/nicoddemus/pytest-plugs/blob/master/.travis.yml
OK. I think you can leave "--use-mirrors" away, btw.
I guess it would be useful to identify which plugins are using just "py.test --help" as smoke test and which ones have real tests, since the former doesn't guarantee that the plugin actually works on that python/pytest combination.
I looked into devpi to simplify the download/unpack/test process and stumbled in a minor roadblock: is there anyway to provide a tox.ini file for those plugins that don't have one? This is the only issue I see right now with using devpi for this particular application.
Now, but i am going to add a "devpi test -c path-to-ini" option today and plan to release devpi-1.2 real soon now. I'll get back here when it's implemented.
Perfect, that would greatly simplify things. Let me know if there is anything I can do to help.
I just implemented it: https://bitbucket.org/hpk42/devpi/commits/3a8ff0503599358deeefedc581e88c03c4... However, it's maybe not exactly what we need. "devpi test PKG" works by looking up the newest PKG release on the index, downloading and unpacking it and then running tox. Passing "devpi test -c tox.ini" will always prefer the external tox.ini, though. We could maybe add a second option "--fallback-ini" option that only is used if there is no tox.ini in the downloaded package. Note that to play with the devpi-{client,server} current trunk you can do: pip install --pre -i https://devpi.net/hpk/dev/+simple/ -U devpi and then run your own devpi-server similar to: http://doc.devpi.net/latest/quickstart-releaseprocess.html And especially try "devpi list PKG" after "devpi test PKG" to see test results.
So, what do you think of the idea of using devpi.net to host test results
and add an endpoint to the API to provide test status images? Or should I create another app to do this?
What do you mean with "test status images"?
In general i am open to extending devpi-server to provide more test-status
info -- just didn't have a chance to think about it in detail yet. I guess we could take the plugin testing bits as a starting point.
I mean like those badge images served by travis that give the current build status, for example:
https://secure.travis-ci.org/hpk42/pytest.png
I think we can follow the same approach, generating a badge that gives the full compatibility status, like a single "py 2.6 2.7 | pytest 2.3.5 | compatible" image, or even multiple images.
But I guess this is too specific to pytest...
Hum, some badge makes sense. Not sure which, though :) holger
Cheers, Bruno.
best, holger
Cheers, Bruno.
I found the reason for that (http vs https) and now the job runs against Python 2.6, 2.7, 3.2 and 3.2.
Thanks! Got back but caught a flue ...
It happens, hope you get better soon. :)
Cheers, Bruno.
On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu> wrote:
> > > > > > - i'd collapse NAME and VERSION columns to save space, i.e. > > > "pytest-bdd-0.6.1". > > > > > > - what about adding download numbers? > > > > > > - as to code organisation: you can leave it as is for now or (maybe > > > better) put all code and generated files into a dedicated
On Mon, Oct 14, 2013 at 23:52 -0300, Bruno Oliveira wrote: directory
> > >
Done, opened a PR for discussion and reviewing. :)
only got to it now, looks good so far!
> > > - we should try to collect repository locations. maybe parsing > > > for github/bitbucket urls would yield most of them automatically? > > > Maybe we need to add some manually. > > > > > > > May I ask why we would need the repository locations? I mean, to work on > > the compatibility feature we can work directly with packages in pypi or > > devpi... > > purely for the person reading the page and wanting to look at the code > or do a PR. >
Oh I see! Well, parsing the urls seems problematic, because we also have to take in account the username ("hpk42/pytest" for example). I looked at the GitHub and BitBucket's rest APIs to see if we could use them for searching, but it seems only GitHub at the moment has a search API (and in beta stage at that).
I think we will have to maintain the repository list manually. Package authors can help as well by simply opening PRs adding repositories for their plugins I guess.
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
> A very interesting bit will be the "2.4.2 compat" and "dev" compat > > <snip> >
Some late night quick hacking and I have come up with this proof of concept:
https://www.travis-ci.org/nicoddemus/pytest-plugs
As it is right now it serves only as a prove of concept, of course. The idea is using travis to handle running things for us, driving it using a script that downloads and run tests using tox. We can then collect json information from that and POST those results back to devpi. Also we can use its build matrix to test against different pytest versions, for example.
Hum, interesting. Is this mainly to make use of travis-ci's build hosts? What is the advantage over just invoking "devpi test PLUGIN-NAME" from the script to automate the download/unpack/test/post-results steps?
I guess we can implement in devpi a concept similar to travis build status imagens, which we could then use in the plugins_index page.
Against python 2.6 the script is failing, I have yet to investigate.
great, i am travelling to Pycon DE tomorrow morning but should be online > from time to time. >
Nice tripe and good PyCon!
Thanks! Got back but caught a flue ...
cheers, holger
Hi Holger, On Thu, Oct 24, 2013 at 11:58 AM, holger krekel <holger@merlinux.eu> wrote:
OK. I think you can leave "--use-mirrors" away, btw.
I usually pass that parameter as is recommended by Travis' docs: "We highly recommend using --use-mirrors if you override dependency installation command to reduce the load on PyPI and possibility of installation failures." (from: http://about.travis-ci.org/docs/user/languages/python/#Travis-CI-uses-pip) Any reason to remove it that I'm not aware of?
Perfect, that would greatly simplify things. Let me know if there is
anything I can do to help.
I just implemented it:
https://bitbucket.org/hpk42/devpi/commits/3a8ff0503599358deeefedc581e88c03c4...
However, it's maybe not exactly what we need. "devpi test PKG" works by looking up the newest PKG release on the index, downloading and unpacking it and then running tox. Passing "devpi test -c tox.ini" will always prefer the external tox.ini, though. We could maybe add a second option "--fallback-ini" option that only is used if there is no tox.ini in the downloaded package.
I guess we could execute that as a two step process, asking for devpi to download the package first to inspect if it has a tox.ini file, and change "devpi test" command line accordingly... but if you think implementing a "--fallback-ini" option is easy/useful for others, that's even better. Note that to play with the devpi-{client,server} current trunk you can do:
pip install --pre -i https://devpi.net/hpk/dev/+simple/ -U devpi
and then run your own devpi-server similar to:
http://doc.devpi.net/latest/quickstart-releaseprocess.html
And especially try "devpi list PKG" after "devpi test PKG" to see test results.
Yes, have been playing around with it, very cool. :)
I think we can follow the same approach, generating a badge that gives the full compatibility status, like a single "py 2.6 2.7 | pytest 2.3.5 | compatible" image, or even multiple images.
But I guess this is too specific to pytest...
Hum, some badge makes sense. Not sure which, though :)
I was thinking that a "pytest compatible" badge is too specific for a generic tool like devpi, don't you think? Creating an app on Heroku to store test results from the travis jobs and serving appropriate badges seems simple enough... is this a direction you think we should take? Cheers, Bruno. holger
Cheers, Bruno.
best, holger
Cheers, Bruno.
I found the reason for that (http vs https) and now the job runs against Python 2.6, 2.7, 3.2 and 3.2.
Thanks! Got back but caught a flue ...
It happens, hope you get better soon. :)
Cheers, Bruno.
On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu
wrote:
On Mon, Oct 14, 2013 at 23:52 -0300, Bruno Oliveira wrote: > > > > > > > > - i'd collapse NAME and VERSION columns to save space,
> > > > "pytest-bdd-0.6.1". > > > > > > > > - what about adding download numbers? > > > > > > > > - as to code organisation: you can leave it as is for now or (maybe > > > > better) put all code and generated files into a dedicated directory > > > > > > > Done, opened a PR for discussion and reviewing. :)
only got to it now, looks good so far!
> > > > - we should try to collect repository locations. maybe
> > > > for github/bitbucket urls would yield most of them automatically? > > > > Maybe we need to add some manually. > > > > > > > > > > May I ask why we would need the repository locations? I mean, to work on > > > the compatibility feature we can work directly with
pypi or > > > devpi... > > > > purely for the person reading the page and wanting to look at
code
> > or do a PR. > > > > Oh I see! Well, parsing the urls seems problematic, because we also have to > take in account the username ("hpk42/pytest" for example). I looked at the > GitHub and BitBucket's rest APIs to see if we could use them for searching, > but it seems only GitHub at the moment has a search API (and in beta stage > at that). > > I think we will have to maintain the repository list manually. Package > authors can help as well by simply opening PRs adding repositories for > their plugins I guess.
What about generating a "plugin details " page for each plugin so we don't have to think too hard about what to put in the index page?
> > A very interesting bit will be the "2.4.2 compat" and "dev" compat > > > <snip> > > > > Some late night quick hacking and I have come up with this
i.e. parsing packages in the proof of
concept: > > https://www.travis-ci.org/nicoddemus/pytest-plugs > > As it is right now it serves only as a prove of concept, of course. The > idea is using travis to handle running things for us, driving it using a > script that downloads and run tests using tox. We can then
collect > > json > > > >> > information from that and POST those results back to devpi. Also we > > can > > > >> use > > > >> > its build matrix to test against different pytest versions, for > > example. > > > >> > > > >> Hum, interesting. Is this mainly to make use of travis-ci's build > > hosts? > > > >> What is the advantage over just invoking "devpi test PLUGIN-NAME" > > from the > > > >> script to automate the download/unpack/test/post-results steps? > > > >> > > > >> > I guess we can implement in devpi a concept similar to travis build > > > >> status > > > >> > imagens, which we could then use in the plugins_index page. > > > >> > > > > >> > Against python 2.6 the script is failing, I have yet to investigate. > > > >> > > > > >> > great, i am travelling to Pycon DE tomorrow morning but should be > > online > > > >> > > from time to time. > > > >> > > > > > >> > > > > >> > Nice tripe and good PyCon! > > > >> > > > >> Thanks! Got back but caught a flue ... > > > >> > > > >> cheers, > > > >> holger > > > >> > > > > > > > > > >
Hi Bruno, On Thu, Oct 24, 2013 at 13:01 -0200, Bruno Oliveira wrote:
Hi Holger,
On Thu, Oct 24, 2013 at 11:58 AM, holger krekel <holger@merlinux.eu> wrote:
OK. I think you can leave "--use-mirrors" away, btw.
I usually pass that parameter as is recommended by Travis' docs:
"We highly recommend using --use-mirrors if you override dependency installation command to reduce the load on PyPI and possibility of installation failures."
(from: http://about.travis-ci.org/docs/user/languages/python/#Travis-CI-uses-pip)
Any reason to remove it that I'm not aware of?
a) official mirrorrs are deprecated, see https://pypi.python.org/mirrors b) PyPI simple pages and release files are served through a CDN
Perfect, that would greatly simplify things. Let me know if there is
anything I can do to help.
I just implemented it:
https://bitbucket.org/hpk42/devpi/commits/3a8ff0503599358deeefedc581e88c03c4...
However, it's maybe not exactly what we need. "devpi test PKG" works by looking up the newest PKG release on the index, downloading and unpacking it and then running tox. Passing "devpi test -c tox.ini" will always prefer the external tox.ini, though. We could maybe add a second option "--fallback-ini" option that only is used if there is no tox.ini in the downloaded package.
I guess we could execute that as a two step process, asking for devpi to download the package first to inspect if it has a tox.ini file, and change "devpi test" command line accordingly... but if you think implementing a "--fallback-ini" option is easy/useful for others, that's even better.
I want to release devpi-1.2 any day now. Adding --fallback-ini as an experimental option is ok, i guess. Once we are through the testing plugin experience we can think about refining it. But others might want to use similar techniques (e.g. Pyramid plugins etc.). So i am going to add the fallback option probably tomorrow.
Note that to play with the devpi-{client,server} current trunk you can do:
pip install --pre -i https://devpi.net/hpk/dev/+simple/ -U devpi
and then run your own devpi-server similar to:
http://doc.devpi.net/latest/quickstart-releaseprocess.html
And especially try "devpi list PKG" after "devpi test PKG" to see test results.
Yes, have been playing around with it, very cool. :)
I think we can follow the same approach, generating a badge that gives the full compatibility status, like a single "py 2.6 2.7 | pytest 2.3.5 | compatible" image, or even multiple images.
But I guess this is too specific to pytest...
Hum, some badge makes sense. Not sure which, though :)
I was thinking that a "pytest compatible" badge is too specific for a generic tool like devpi, don't you think?
Creating an app on Heroku to store test results from the travis jobs and serving appropriate badges seems simple enough... is this a direction you think we should take?
Might make sense. OTOH test results are already stored with devpi so at least it makes sense to try to use that information. Might need a bit of web API tweeks to make usage convenient. best, holger
Cheers, Bruno.
holger
Cheers, Bruno.
best, holger
Cheers, Bruno.
I found the reason for that (http vs https) and now the job runs against Python 2.6, 2.7, 3.2 and 3.2.
Thanks! Got back but caught a flue ... >
It happens, hope you get better soon. :)
Cheers, Bruno.
On Sun, Oct 20, 2013 at 3:52 AM, holger krekel <holger@merlinux.eu
wrote:
> On Mon, Oct 14, 2013 at 23:52 -0300, Bruno Oliveira wrote: > > > > > > > > > > - i'd collapse NAME and VERSION columns to save space,
> > > > > "pytest-bdd-0.6.1". > > > > > > > > > > - what about adding download numbers? > > > > > > > > > > - as to code organisation: you can leave it as is for now or > (maybe > > > > > better) put all code and generated files into a dedicated > directory > > > > > > > > > > > Done, opened a PR for discussion and reviewing. :) > > only got to it now, looks good so far! > > > > > > - we should try to collect repository locations. maybe
> > > > > for github/bitbucket urls would yield most of them > automatically? > > > > > Maybe we need to add some manually. > > > > > > > > > > > > > May I ask why we would need the repository locations? I mean, to > work on > > > > the compatibility feature we can work directly with
> pypi or > > > > devpi... > > > > > > purely for the person reading the page and wanting to look at
code
> > > or do a PR. > > > > > > > Oh I see! Well, parsing the urls seems problematic, because we also > have to > > take in account the username ("hpk42/pytest" for example). I looked at > the > > GitHub and BitBucket's rest APIs to see if we could use them for > searching, > > but it seems only GitHub at the moment has a search API (and in beta > stage > > at that). > > > > I think we will have to maintain the repository list manually. Package > > authors can help as well by simply opening PRs adding repositories for > > their plugins I guess. > > What about generating a "plugin details " page for each plugin so we don't > have to think too hard about what to put in the index page? > > > > A very interesting bit will be the "2.4.2 compat" and "dev" compat > > > > <snip> > > > > > > > Some late night quick hacking and I have come up with this
i.e. parsing packages in the proof of
> concept: > > > > https://www.travis-ci.org/nicoddemus/pytest-plugs > > > > As it is right now it serves only as a prove of concept, of course. The > > idea is using travis to handle running things for us, driving it using a > > script that downloads and run tests using tox. We can then
collect > > json > > > >> > information from that and POST those results back to devpi. Also we > > can > > > >> use > > > >> > its build matrix to test against different pytest versions, for > > example. > > > >> > > > >> Hum, interesting. Is this mainly to make use of travis-ci's build > > hosts? > > > >> What is the advantage over just invoking "devpi test PLUGIN-NAME" > > from the > > > >> script to automate the download/unpack/test/post-results steps? > > > >> > > > >> > I guess we can implement in devpi a concept similar to travis build > > > >> status > > > >> > imagens, which we could then use in the plugins_index page. > > > >> > > > > >> > Against python 2.6 the script is failing, I have yet to investigate. > > > >> > > > > >> > great, i am travelling to Pycon DE tomorrow morning but should be > > online > > > >> > > from time to time. > > > >> > > > > > >> > > > > >> > Nice tripe and good PyCon! > > > >> > > > >> Thanks! Got back but caught a flue ... > > > >> > > > >> cheers, > > > >> holger > > > >> > > > > > > > > > >
On Thu, Oct 24, 2013 at 20:39 +0000, holger krekel wrote:
I guess we could execute that as a two step process, asking for devpi to download the package first to inspect if it has a tox.ini file, and change "devpi test" command line accordingly... but if you think implementing a "--fallback-ini" option is easy/useful for others, that's even better.
I want to release devpi-1.2 any day now. Adding --fallback-ini as an experimental option is ok, i guess. Once we are through the testing plugin experience we can think about refining it. But others might want to use similar techniques (e.g. Pyramid plugins etc.). So i am going to add the fallback option probably tomorrow.
I just did devpi-*-1.2 releases. If you install devpi-client>=1.2 you get "devpi test --fallback-ini someini". Hope it helps. best, holger
It certainly helps! :) Cheers, Bruno On Wed, Oct 30, 2013 at 8:42 AM, holger krekel <holger@merlinux.eu> wrote:
On Thu, Oct 24, 2013 at 20:39 +0000, holger krekel wrote:
I guess we could execute that as a two step process, asking for devpi to download the package first to inspect if it has a tox.ini file, and change "devpi test" command line accordingly... but if you think implementing a "--fallback-ini" option is easy/useful for others, that's even better.
I want to release devpi-1.2 any day now. Adding --fallback-ini as an experimental option is ok, i guess. Once we are through the testing plugin experience we can think about refining it. But others might want to use similar techniques (e.g. Pyramid plugins etc.). So i am going to add the fallback option probably tomorrow.
I just did devpi-*-1.2 releases. If you install devpi-client>=1.2 you get "devpi test --fallback-ini someini". Hope it helps.
best, holger
Hi Holger, all, Back from vacation, so I'm back to this. :) On Wed, Oct 30, 2013 at 8:42 AM, holger krekel <holger@merlinux.eu> wrote:
I just did devpi-*-1.2 releases. If you install devpi-client>=1.2 you get "devpi test --fallback-ini someini". Hope it helps.
Works great, thanks. I'm keeping a branch "devpi" on pytest-plugs for this. You can see one of the build results here: https://travis-ci.org/nicoddemus/pytest-plugs/builds/13596968 It basically installs a devpi server using "quickstart" and issues a series of "devpi test" commands. Neat. One question: 1) Is there any way to overwrite the pytest version to use for running "devpi test"? This is required if we want to test against different pytest versions. Cheers, Bruno.
best, holger
Hi Bruno, On Wed, Nov 06, 2013 at 17:22 -0200, Bruno Oliveira wrote:
Hi Holger, all,
Back from vacation, so I'm back to this. :)
On Wed, Oct 30, 2013 at 8:42 AM, holger krekel <holger@merlinux.eu> wrote:
I just did devpi-*-1.2 releases. If you install devpi-client>=1.2 you get "devpi test --fallback-ini someini". Hope it helps.
Works great, thanks. I'm keeping a branch "devpi" on pytest-plugs for this. You can see one of the build results here:
https://travis-ci.org/nicoddemus/pytest-plugs/builds/13596968
It basically installs a devpi server using "quickstart" and issues a series of "devpi test" commands. Neat.
One question:
1) Is there any way to overwrite the pytest version to use for running "devpi test"? This is required if we want to test against different pytest versions.
From "devpi test" there is no way currently other than passing in a custom tox.ini. We could think about adding an option to tox but it is not clear how that would look like. Maybe we could assume that the tox.ini that is being used has certain environments that would pin a pytest version. It is then a matter of reading out the "toxresults" that are posted to devpi after the tests have run. (you also see the result of reading that out from "devpi list X". There is no API documentation how to read it out but the code doing it is contained in devpi_client/list_remove.py (it's basically performing some http json requests). Our own "--fallback-ini" could of course pin pytest versions as well. best, holger
Cheers, Bruno.
best, holger
Hi Holger, On Fri, Nov 8, 2013 at 4:04 AM, holger krekel <holger@merlinux.eu> wrote:
From "devpi test" there is no way currently other than passing in a custom tox.ini. We could think about adding an option to tox but it is not clear how that would look like. Maybe we could assume that the tox.ini that is being used has certain environments that would pin a pytest version. It is then a matter of reading out the "toxresults" that are posted to devpi after the tests have run. (you also see the result of reading that out from "devpi list X". There is no API documentation how to read it out but the code doing it is contained in devpi_client/list_remove.py (it's basically performing some http json requests). Our own "--fallback-ini" could of course pin pytest versions as well.
Hmm, I just realized that tox uses the latest pytest version, not the one I have installed in the environment it is running (which makes sense). This means that the travis jobs for pytest 2.3.5 are actually running the tests using 2.4.2. :) To solve this, I guess the simplest approach would be to read the tox.ini file and replace occurrences of plain "pytest" to "pytest==<version>" to enforce tox to use the version we want. Don't know how/if that would play out nicely with devpi, seems very specific for the purpose we are trying to accomplish. I'm finishing the page where the results are posted, at the very least we will have a compatibility matrix of plugins against python versions and the latest pytest version. Once I finish this (probably today) I will let you know so you can take a look. Best Regards, Bruno.
Cheers, Bruno.
best, holger
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBAgAGBQJSfH7gAAoJEI47A6J5t3LWypAH+wU9c8FtDhrZqZQ+cZ33dgvD meYzYwKPsuZ8Xe/OGRPteroD/PugdYy1ewPg3C7VPLdKHMsL1FLR89D4rY3yqjX0 CVS7f9O1mQaauQ83bB0F/YwMajZgcahoUyIZn5q5GSdTr+DWHqUhvIyJtJ4ykysK 5MvaVBkSEj8Bx1ZGdOQtWSRo0CdURceQ8yV3jm9z24L8VlEOpAr3tCvkOhNyOcZN aj71mZ34hvXFjLFhRgGj3pYbU7BHumtWYPGcyCnxh2/anpX9azU3tMin1IU4Sn8L EH5Vorcx4bg80tDu9gOLQHcO1mro1MKPqeinoITDG4nJLndi9SM9GaZSq2+jilE= =GXi4 -----END PGP SIGNATURE-----
On Fri, Nov 8, 2013 at 10:58 AM, Bruno Oliveira <nicoddemus@gmail.com>wrote:
I'm finishing the page where the results are posted, at the very least we will have a compatibility matrix of plugins against python versions and the latest pytest version. Once I finish this (probably today) I will let you know so you can take a look.
I just finished a first version, please take a look: http://pytest-plugs.herokuapp.com/ Also, an endpoint "/status/<plugin>" is available to query status images (for an example, please see http://pytest-plugs.herokuapp.com/status/pytest-blockage-0.1?py=py27&pytest=2.4.2). I'm thinking we can use those URLs to embed status images for the latest pytest version into the plugin index page, similar to the summary table at the start of the pytest-plugs heroku app. As I noted earlier, the tests reported for pytest-2.3.5 are actually for 2.4.2 since I didn't overwrite the environment in the tox.ini files. I will try the approach of replacing the "pytest" dependency manually, unless you have another suggestion. This is the very first version, so comments are very welcome. Cheers,
Best Regards, Bruno.
Cheers, Bruno.
best, holger
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBAgAGBQJSfH7gAAoJEI47A6J5t3LWypAH+wU9c8FtDhrZqZQ+cZ33dgvD meYzYwKPsuZ8Xe/OGRPteroD/PugdYy1ewPg3C7VPLdKHMsL1FLR89D4rY3yqjX0 CVS7f9O1mQaauQ83bB0F/YwMajZgcahoUyIZn5q5GSdTr+DWHqUhvIyJtJ4ykysK 5MvaVBkSEj8Bx1ZGdOQtWSRo0CdURceQ8yV3jm9z24L8VlEOpAr3tCvkOhNyOcZN aj71mZ34hvXFjLFhRgGj3pYbU7BHumtWYPGcyCnxh2/anpX9azU3tMin1IU4Sn8L EH5Vorcx4bg80tDu9gOLQHcO1mro1MKPqeinoITDG4nJLndi9SM9GaZSq2+jilE= =GXi4 -----END PGP SIGNATURE-----
Hi Bruno, On Fri, Nov 08, 2013 at 19:28 -0200, Bruno Oliveira wrote:
On Fri, Nov 8, 2013 at 10:58 AM, Bruno Oliveira <nicoddemus@gmail.com>wrote:
I'm finishing the page where the results are posted, at the very least we will have a compatibility matrix of plugins against python versions and the latest pytest version. Once I finish this (probably today) I will let you know so you can take a look.
I just finished a first version, please take a look:
looks good on first sight!
Also, an endpoint "/status/<plugin>" is available to query status images (for an example, please see http://pytest-plugs.herokuapp.com/status/pytest-blockage-0.1?py=py27&pytest=2.4.2).
Works.
I'm thinking we can use those URLs to embed status images for the latest pytest version into the plugin index page, similar to the summary table at the start of the pytest-plugs heroku app.
As I noted earlier, the tests reported for pytest-2.3.5 are actually for 2.4.2 since I didn't overwrite the environment in the tox.ini files. I will try the approach of replacing the "pytest" dependency manually, unless you have another suggestion.
Are the tests run via creating a ``tox.ini`` yourself, overriding any tox.ini that might be contained with the package? In any case, it would be useful to everyone (plugin authors, users, pytest core devs) to see what is actually breaking (or working), i.e. the actual test failure/success log. Is that available somewhere already? cheers, holger
This is the very first version, so comments are very welcome.
Cheers,
Best Regards, Bruno.
Cheers, Bruno.
best, holger
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBAgAGBQJSfH7gAAoJEI47A6J5t3LWypAH+wU9c8FtDhrZqZQ+cZ33dgvD meYzYwKPsuZ8Xe/OGRPteroD/PugdYy1ewPg3C7VPLdKHMsL1FLR89D4rY3yqjX0 CVS7f9O1mQaauQ83bB0F/YwMajZgcahoUyIZn5q5GSdTr+DWHqUhvIyJtJ4ykysK 5MvaVBkSEj8Bx1ZGdOQtWSRo0CdURceQ8yV3jm9z24L8VlEOpAr3tCvkOhNyOcZN aj71mZ34hvXFjLFhRgGj3pYbU7BHumtWYPGcyCnxh2/anpX9azU3tMin1IU4Sn8L EH5Vorcx4bg80tDu9gOLQHcO1mro1MKPqeinoITDG4nJLndi9SM9GaZSq2+jilE= =GXi4 -----END PGP SIGNATURE-----
Hi Holger, all, On Tue, Nov 12, 2013 at 7:25 AM, holger krekel <holger@merlinux.eu> wrote:
I'm thinking we can use those URLs to embed status images for the latest pytest version into the plugin index page, similar to the summary table at the start of the pytest-plugs heroku app.
As I noted earlier, the tests reported for pytest-2.3.5 are actually for 2.4.2 since I didn't overwrite the environment in the tox.ini files. I will try the approach of replacing the "pytest" dependency manually, unless you have another suggestion.
Are the tests run via creating a ``tox.ini`` yourself, overriding any tox.ini that might be contained with the package?
I'm creating a tox.ini file if there is none, or if there is already a tox.ini inplace I try to use a simple regex to replace "pytest" occurrences to "pytest==$PYTEST_VERSION" in an attempt to enforce the pytest version that should be used for running the tests. This has worked for most plugins, but some of them setup the dependencies in a slightly different manner than what my simple regex-replace trick could handle: * Some packages declare a specific pytest a version as dependency, like "pytest==2.4.0" or "pytest>=2.0". * One of the packages installed the dependencies from a requirements file, using "-r requirements-tests.txt" instead of listing the dependencies on the tox.ini file itself. I could of course make my regex more complex in an attempt to cover all the cases, but I feel this would mean a lot of duplicated work that is already done by tox itself. Any suggestions? In any case, it would be useful to everyone (plugin authors, users,
pytest core devs) to see what is actually breaking (or working), i.e. the actual test failure/success log. Is that available somewhere already?
Hmm good point. It is available from the travis-ci output log, but that is not very friendly. I will change the travis-ci jobs to also post the results from "tox --result-json" which contains all the details from that test session, and make that available from the webpage as well. So I'm thinking this should be the next steps: 1. Update plugins index page to include the status images; 2. Post test results to the pytest-plugs page and make it available from there; 3. Find a nice way to ensure we are running "tox" with the pytest version we want; About 3), I just had an idea... what if we install a devpi server that would serve only the pytest version we want (say, 2.3.5) and force tox to use that devpi server by using the "-i" option? Would that be possible? ("Shadowed Release Files" seems to be what would be required for this to work). Cheers, Bruno.
Hi Bruno, On Tue, Nov 12, 2013 at 22:51 -0200, Bruno Oliveira wrote:
Hi Holger, all,
On Tue, Nov 12, 2013 at 7:25 AM, holger krekel <holger@merlinux.eu> wrote:
I'm thinking we can use those URLs to embed status images for the latest pytest version into the plugin index page, similar to the summary table at the start of the pytest-plugs heroku app.
As I noted earlier, the tests reported for pytest-2.3.5 are actually for 2.4.2 since I didn't overwrite the environment in the tox.ini files. I will try the approach of replacing the "pytest" dependency manually, unless you have another suggestion.
Are the tests run via creating a ``tox.ini`` yourself, overriding any tox.ini that might be contained with the package?
I'm creating a tox.ini file if there is none, or if there is already a tox.ini inplace I try to use a simple regex to replace "pytest" occurrences to "pytest==$PYTEST_VERSION" in an attempt to enforce the pytest version that should be used for running the tests. This has worked for most plugins, but some of them setup the dependencies in a slightly different manner than what my simple regex-replace trick could handle:
* Some packages declare a specific pytest a version as dependency, like "pytest==2.4.0" or "pytest>=2.0". * One of the packages installed the dependencies from a requirements file, using "-r requirements-tests.txt" instead of listing the dependencies on the tox.ini file itself.
I could of course make my regex more complex in an attempt to cover all the cases, but I feel this would mean a lot of duplicated work that is already done by tox itself. Any suggestions?
I think we should allow instructing tox to override a version for a dependency. Something like tox --force-dep-version=pytest==2.4.2 This would make tox replace any "pytest" dependency that is specified with tox itself. It would not work for implicit dependencies specified within a setup.py's install_requires setting. I guess we should extend the result-json format to include a list of all packages and dependencies for each environment so that we can do a double-check after tests were run.
In any case, it would be useful to everyone (plugin authors, users,
pytest core devs) to see what is actually breaking (or working), i.e. the actual test failure/success log. Is that available somewhere already?
Hmm good point. It is available from the travis-ci output log, but that is not very friendly. I will change the travis-ci jobs to also post the results from "tox --result-json" which contains all the details from that test session, and make that available from the webpage as well.
So I'm thinking this should be the next steps:
1. Update plugins index page to include the status images; 2. Post test results to the pytest-plugs page and make it available from there; 3. Find a nice way to ensure we are running "tox" with the pytest version we want;
makes sense.
About 3), I just had an idea... what if we install a devpi server that would serve only the pytest version we want (say, 2.3.5) and force tox to use that devpi server by using the "-i" option? Would that be possible? ("Shadowed Release Files" seems to be what would be required for this to work).
See my suggestion above. I think it would be more effort to do it indirectly via devpi-server here. cheers, holger
Cheers, Bruno.
Hi Holger, I think we should allow instructing tox to override
a version for a dependency. Something like
tox --force-dep-version=pytest==2.4.2
This would make tox replace any "pytest" dependency that is specified with tox itself. It would not work for implicit dependencies specified within a setup.py's install_requires setting. I guess we should extend the result-json format to include a list of all packages and dependencies for each environment so that we can do a double-check after tests were run.
I agree, that seems cleaner. I will take a look at tox in order to open a PR for that.
In any case, it would be useful to everyone (plugin authors, users,
pytest core devs) to see what is actually breaking (or working), i.e. the actual test failure/success log. Is that available somewhere already?
Hmm good point. It is available from the travis-ci output log, but that is not very friendly. I will change the travis-ci jobs to also post the results from "tox --result-json" which contains all the details from that test session, and make that available from the webpage as well.
So I'm thinking this should be the next steps:
1. Update plugins index page to include the status images; 2. Post test results to the pytest-plugs page and make it available from there; 3. Find a nice way to ensure we are running "tox" with the pytest version we want;
makes sense.
About 3), I just had an idea... what if we install a devpi server that would serve only the pytest version we want (say, 2.3.5) and force tox to use that devpi server by using the "-i" option? Would that be possible? ("Shadowed Release Files" seems to be what would be required for this to work).
See my suggestion above. I think it would be more effort to do it indirectly via devpi-server here.
OK, let's see how that goes. :) Cheers, Bruno
Hi Bruno, On Wed, Nov 13, 2013 at 12:30 -0200, Bruno Oliveira wrote:
Hi Holger,
I think we should allow instructing tox to override
a version for a dependency. Something like
tox --force-dep-version=pytest==2.4.2
This would make tox replace any "pytest" dependency that is specified with tox itself. It would not work for implicit dependencies specified within a setup.py's install_requires setting. I guess we should extend the result-json format to include a list of all packages and dependencies for each environment so that we can do a double-check after tests were run.
I agree, that seems cleaner. I will take a look at tox in order to open a PR for that.
great, thanks! And let me know if you get stuck on that or need help. cheers, holger
In any case, it would be useful to everyone (plugin authors, users,
pytest core devs) to see what is actually breaking (or working), i.e. the actual test failure/success log. Is that available somewhere already?
Hmm good point. It is available from the travis-ci output log, but that is not very friendly. I will change the travis-ci jobs to also post the results from "tox --result-json" which contains all the details from that test session, and make that available from the webpage as well.
So I'm thinking this should be the next steps:
1. Update plugins index page to include the status images; 2. Post test results to the pytest-plugs page and make it available from there; 3. Find a nice way to ensure we are running "tox" with the pytest version we want;
makes sense.
About 3), I just had an idea... what if we install a devpi server that would serve only the pytest version we want (say, 2.3.5) and force tox to use that devpi server by using the "-i" option? Would that be possible? ("Shadowed Release Files" seems to be what would be required for this to work).
See my suggestion above. I think it would be more effort to do it indirectly via devpi-server here.
OK, let's see how that goes. :)
Cheers, Bruno
participants (3)
-
Bruno Oliveira -
Florian Schulze -
holger krekel