Hi,
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it? ---------------
1. Join #python-dev on irc.freenode.net. 2. Ask for test privilege from any one of kushal,Taggnostr,bitdancer 3. They will issue a simple command. #add: YOUR_NICK_NAME 4. You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
This will do the following: Start a new job on ci.centos.org, announce it on the channel, and announce the result also.
I will be working on a minimal lint for patches, and include it the workflow.
The current steps can be found at [2]. Each build is happening on a fresh system.
Limitations -----------
1. It excepts one single patch to contain all the required changes (instead of a series of patches). 2. It runs only on x86_64 architecture, CentOS7 based systems.
[1] https://ci.centos.org/job/cPython-build-patch/ [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh
Kushal
In the airport but I wanted to say thanks for this!
On Tue, May 19, 2015, 11:03 Kushal Das kushaldas@gmail.com wrote:
Hi,
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it?
Join #python-dev on irc.freenode.net.
Ask for test privilege from any one of kushal,Taggnostr,bitdancer
They will issue a simple command. #add: YOUR_NICK_NAME
You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
This will do the following: Start a new job on ci.centos.org, announce it on the channel, and announce the result also.
I will be working on a minimal lint for patches, and include it the workflow.
The current steps can be found at [2]. Each build is happening on a fresh system.
Limitations
- It excepts one single patch to contain all the required changes
(instead of a series of patches). 2. It runs only on x86_64 architecture, CentOS7 based systems.
[1] https://ci.centos.org/job/cPython-build-patch/ [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh
Kushal
Fedora Cloud Engineer CPython Core Developer Director @ Python Software Foundation http://kushaldas.in _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On Tue, May 19, 2015 at 6:02 PM, Kushal Das kushaldas@gmail.com wrote:
Hi,
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it?
Join #python-dev on irc.freenode.net.
Ask for test privilege from any one of kushal,Taggnostr,bitdancer
They will issue a simple command. #add: YOUR_NICK_NAME
You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
This will do the following: Start a new job on ci.centos.org, announce it on the channel, and announce the result also.
Hi Kushal,
Looks great, thanks! :)
Two comments:
* It would be good to have a pypatcher repository at hg.python.org (at least a mirror), so we can work on it together without dealing with "add me to the repo" messages on GitHub. * Do you have a roadmap or a TODO list? For example, I think downloading a tarball of the default branch every time (or is it cached?) would be a little bit slow. Do you have a plan to make the workflow Mercurial based (e.g. "hg pull -u, hg imp --no-c issueXXXX.diff, compile" instead of "wget tarball, extract it, apply patch, compile")?
I will be working on a minimal lint for patches, and include it the workflow.
Could you give more details about the linter? Can we use Tools/scripts/patchcheck.py?
Thanks!
--Berker
On 19/05/15, Berker Peksağ wrote:
On Tue, May 19, 2015 at 6:02 PM, Kushal Das kushaldas@gmail.com wrote:
Hi,
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it?
Join #python-dev on irc.freenode.net.
Ask for test privilege from any one of kushal,Taggnostr,bitdancer
They will issue a simple command. #add: YOUR_NICK_NAME
You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
This will do the following: Start a new job on ci.centos.org, announce it on the channel, and announce the result also.
Hi Kushal,
Looks great, thanks! :)
Two comments:
- It would be good to have a pypatcher repository at hg.python.org (at
least a mirror), so we can work on it together without dealing with "add me to the repo" messages on GitHub.
We can surely do this. I started with github as generally most people are already there. Do you know what is the procedure for creating a new repo in hg.python.org?
- Do you have a roadmap or a TODO list? For example, I think
downloading a tarball of the default branch every time (or is it cached?) would be a little bit slow. Do you have a plan to make the workflow Mercurial based (e.g. "hg pull -u, hg imp --no-c issueXXXX.diff, compile" instead of "wget tarball, extract it, apply patch, compile")?
I will have to work on the TODO list, I will post it on the repo itself. The downloading tarball currently takes around 15-16 seconds, which I found fast enough to start with. I personally always use standard patch command, that is why I chose this approach instead of hg. We can always improve the workflow :)
I will be working on a minimal lint for patches, and include it the workflow.
Could you give more details about the linter? Can we use Tools/scripts/patchcheck.py?
For this I really never thought much. We should discuss more on this to find out what all we can do.
Kushal
Cool! Thanks!
BuildBot integration?
* http://docs.buildbot.net/latest/full.html#source-stamps * http://docs.buildbot.net/latest/full.html#enabling-the-irc-bot * http://docs.buildbot.net/latest/full.html#choosing-a-change-source
https://github.com/audreyr/cookiecutter-pypackage (requirements.txt, setup.py, Makefile)
pip install cookiecutter cookiecutter gh:audreyr/cookiecutter-pypackage
On Tue, May 19, 2015 at 10:02 AM, Kushal Das kushaldas@gmail.com wrote:
Hi,
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it?
Join #python-dev on irc.freenode.net.
Ask for test privilege from any one of kushal,Taggnostr,bitdancer
They will issue a simple command. #add: YOUR_NICK_NAME
You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
This will do the following: Start a new job on ci.centos.org, announce it on the channel, and announce the result also.
I will be working on a minimal lint for patches, and include it the workflow.
The current steps can be found at [2]. Each build is happening on a fresh system.
Limitations
- It excepts one single patch to contain all the required changes
(instead of a series of patches). 2. It runs only on x86_64 architecture, CentOS7 based systems.
[1] https://ci.centos.org/job/cPython-build-patch/ [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh
Kushal
Fedora Cloud Engineer CPython Core Developer Director @ Python Software Foundation http://kushaldas.in _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
http://docs.buildbot.net/latest/search.html?q=docker ...
Here's a BuildBot Dockerfile: http://docs.buildbot.net/latest/manual/cfg-buildslaves-docker.html#image-cre...
On Tue, May 19, 2015 at 1:22 PM, Wes Turner wes.turner@gmail.com wrote:
Cool! Thanks!
BuildBot integration?
- http://docs.buildbot.net/latest/full.html#source-stamps
- http://docs.buildbot.net/latest/full.html#enabling-the-irc-bot
- http://docs.buildbot.net/latest/full.html#choosing-a-change-source
https://github.com/audreyr/cookiecutter-pypackage (requirements.txt, setup.py, Makefile)
pip install cookiecutter cookiecutter gh:audreyr/cookiecutter-pypackage
On Tue, May 19, 2015 at 10:02 AM, Kushal Das kushaldas@gmail.com wrote:
Hi,
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it?
Join #python-dev on irc.freenode.net.
Ask for test privilege from any one of kushal,Taggnostr,bitdancer
They will issue a simple command. #add: YOUR_NICK_NAME
You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
This will do the following: Start a new job on ci.centos.org, announce it on the channel, and announce the result also.
I will be working on a minimal lint for patches, and include it the workflow.
The current steps can be found at [2]. Each build is happening on a fresh system.
Limitations
- It excepts one single patch to contain all the required changes
(instead of a series of patches). 2. It runs only on x86_64 architecture, CentOS7 based systems.
[1] https://ci.centos.org/job/cPython-build-patch/ [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh
Kushal
Fedora Cloud Engineer CPython Core Developer Director @ Python Software Foundation http://kushaldas.in _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
Here's a GerritChangeSource for BuildBot events: http://docs.buildbot.net/latest/full.html#chsrc-GerritChangeSource
Great idea, thanks again!
On Tue, May 19, 2015 at 1:25 PM, Wes Turner wes.turner@gmail.com wrote:
http://docs.buildbot.net/latest/search.html?q=docker ...
Here's a BuildBot Dockerfile: http://docs.buildbot.net/latest/manual/cfg-buildslaves-docker.html#image-cre...
On Tue, May 19, 2015 at 1:22 PM, Wes Turner wes.turner@gmail.com wrote:
Cool! Thanks!
BuildBot integration?
- http://docs.buildbot.net/latest/full.html#source-stamps
- http://docs.buildbot.net/latest/full.html#enabling-the-irc-bot
- http://docs.buildbot.net/latest/full.html#choosing-a-change-source
https://github.com/audreyr/cookiecutter-pypackage (requirements.txt, setup.py, Makefile)
pip install cookiecutter cookiecutter gh:audreyr/cookiecutter-pypackage
On Tue, May 19, 2015 at 10:02 AM, Kushal Das kushaldas@gmail.com wrote:
Hi,
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it?
Join #python-dev on irc.freenode.net.
Ask for test privilege from any one of kushal,Taggnostr,bitdancer
They will issue a simple command. #add: YOUR_NICK_NAME
You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
This will do the following: Start a new job on ci.centos.org, announce it on the channel, and announce the result also.
I will be working on a minimal lint for patches, and include it the workflow.
The current steps can be found at [2]. Each build is happening on a fresh system.
Limitations
- It excepts one single patch to contain all the required changes
(instead of a series of patches). 2. It runs only on x86_64 architecture, CentOS7 based systems.
[1] https://ci.centos.org/job/cPython-build-patch/ [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh
Kushal
Fedora Cloud Engineer CPython Core Developer Director @ Python Software Foundation http://kushaldas.in _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com
On 5/19/2015 11:02 AM, Kushal Das wrote:
Hi,
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it?
Join #python-dev on irc.freenode.net.
Ask for test privilege from any one of kushal,Taggnostr,bitdancer
They will issue a simple command. #add: YOUR_NICK_NAME
You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
What if there are multiple patches on the issue? Pick the latest? This is not correct if someone follows up a patch with a 2.7 backport, or if there are competing patches.
This will do the following: Start a new job on ci.centos.org, announce it on the channel, and announce the result also.
I will be working on a minimal lint for patches, and include it the workflow.
The current steps can be found at [2]. Each build is happening on a fresh system.
Limitations
- It excepts one single patch to contain all the required changes
(instead of a series of patches).
- It runs only on x86_64 architecture, CentOS7 based systems.
[1] https://ci.centos.org/job/cPython-build-patch/ [2] https://github.com/kushaldas/pypatcher/blob/master/pypatcher.sh
Kushal
On Tue, May 19, 2015 at 11:37 PM, Terry Reedy tjreedy@udel.edu wrote:
On 5/19/2015 11:02 AM, Kushal Das wrote:
Hi,
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it?
Join #python-dev on irc.freenode.net.
Ask for test privilege from any one of kushal,Taggnostr,bitdancer
They will issue a simple command. #add: YOUR_NICK_NAME
You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
What if there are multiple patches on the issue? Pick the latest? This is not correct if someone follows up a patch with a 2.7 backport, or if there are competing patches.
Here is the code that checks how much outstanding patches a certain module has by downloading all patches from open issues, parsing them and comparing the paths. It is possible to reuse the parser to check paths in patch against paths present in certain Python versions, or add different heuristics.
https://bitbucket.org/techtonik/python-stdlib
All this is pure Python and should work cross-platform too.
This was intended to add status for bugs.python.org, but the work on Roundup had stalled due to uncertainty and despair on how to handle utf-8 (internal to Roundup) vs unicode (internal to Jinja2) in this issue: http://issues.roundup-tracker.org/issue2550811 The root of the problem is that Python 2.7 uses 'ascii' and not 'utf-8' internally, so Jinja2 engine fails with 'ascii' ordinal not in range somewhere in the way. Need an expert advice how to handle that, because my brain power is not enough to process it.
On 5/20/2015 3:44 AM, anatoly techtonik wrote:
This was intended to add status for bugs.python.org, but the work on Roundup had stalled due to uncertainty and despair on how to handle utf-8 (internal to Roundup) vs unicode (internal to Jinja2) in this issue: http://issues.roundup-tracker.org/issue2550811 The root of the problem is that Python 2.7 uses 'ascii' and not 'utf-8' internally, so Jinja2 engine fails with 'ascii' ordinal not in range somewhere in the way. Need an expert advice how to handle that, because my brain power is not enough to process it.
In my view, the root of the problem is using Python 2 and working with encoded bytes. The fix is to upgrade to Python 3.3+, with an improved text model (unicode str class), and work with text. Follow the standard process: decode encoded bytes to text when received from browsers, work internally with text, and encode output just before sending to browsers.
On 5/19/15, Terry Reedy tjreedy@udel.edu wrote:
On 5/19/2015 11:02 AM, Kushal Das wrote:
Hi,
Hi !
I'm not very familiar with python-dev development workflows . Nonetheless I just wanted to mention something that proved to be useful for me in the past .
With the help of CentOS project I am happy to announce an automated system [1] to test patches from bugs.python.org. This can be fully automated to test the patches whenever someone uploads a patch in the roundup, but for now it accepts IRC commands on #python-dev channel. I worked on a docker based prototype during sprints in PyCon.
How to use it?
Join #python-dev on irc.freenode.net.
Ask for test privilege from any one of kushal,Taggnostr,bitdancer
They will issue a simple command. #add: YOUR_NICK_NAME
You can then test by issuing the following command in the channel:
#test: BUGNUMBER like #test: 21271
What if there are multiple patches on the issue? Pick the latest? This is not correct if someone follows up a patch with a 2.7 backport, or if there are competing patches.
[...]
It is a fact that running automated tests for patches is a really useful feature . Nevertheless , IMHO for this to succeed at large scale there is a need to manage the content of patches themselves , the base version they were built upon , as well as their order should they be stacked . My suggestion for you therefore is to use Hg patch repositories [1]_ as the starting point for your patch CI system . Some of the benefits I could mention :
- triggering (patch) builds on commit via web hooks - CI infrastructure needed turns out to be very similar to the one setup for the main project - Commands to work on patch queue repositories are easy to learn - The possibility of editing series file is also useful for ignoring some patches without removing their contents . - halt if patch cannot be applied upon latest version * ... but still be able to see it in action by checking out the right version of the code base used to build it in first place . - try the patch against different versions of the code base as it evolves - fuzzy refresh - version control for patches - multiple branches * which may be bound to tickets in many ways e.g. naming conventions * ... particularly useful for competing patches .
There are a few black spots too . Patch repositories deal with the diff of a diff , hence some operations applied upon patches (e.g. merging) might be quite messy , Most of the time this is no big deal though .
The following are repositories I used while developing Apache Bloodhound , during incubation and after it became a TLP . I'm including them to illustrate branching and naming conventions (I used) to keep track of tickets .
https://bitbucket.org/olemis/bloodhound-incubator-mq/ https://bitbucket.org/olemis/bloodhound-mq
HTH , since this way the workflow would be tightly integrated with Mercurial , as highlighted by Berker Peksağ in previous messages .
.. [1] http://mercurial.selenic.com/wiki/MqTutorial
-
anatoly techtonik -
Berker Peksağ -
Brett Cannon -
Kushal Das -
Olemis Lang -
Terry Reedy -
Wes Turner