http://hg.python.org/devguide/rev/74f6a2fea6d1 changeset: 407:74f6a2fea6d1 user: Antoine Pitrou <solipsis@pitrou.net> date: Wed Mar 30 22:59:11 2011 +0200 summary: Add a page about continuous integration (buildbots) files: buildbots.rst | 177 ++++++++++++++++++++++++++++++++++++++ index.rst | 10 +- 2 files changed, 183 insertions(+), 4 deletions(-) diff --git a/buildbots.rst b/buildbots.rst new file mode 100644 --- /dev/null +++ b/buildbots.rst @@ -0,0 +1,177 @@ +.. _buildbots: + +Continuous Integration +====================== + +To assert that there are no regressions in the :doc:`development and maintenance +branches <devcycle>`, Python has a set of dedicated machines (called *buildbots* or +*build slaves*) used for continuous integration. They span a number of +hardware/operating system combinations. Furthermore, each machine hosts +several *builders*, one per active branch: when a new change is pushed +to this branch on the public Mercurial repository, all corresponding builders +will schedule a new build to be run as soon as possible. + +The build steps run by the buildbots are the following: + +* Checkout of the source tree for the changeset which triggered the build +* Compiling Python +* Running the test suite using :ref:`strenuous settings <strenuous_testing>` +* Cleaning up the build tree + +It is your responsibility, as a core developer, to check the automatic +build results after you push a change to the repository. It is therefore +important that you get acquainted with the way these results are presented, +and how various kinds of failures can be explained and diagnosed. + +Checking results of automatic builds +------------------------------------ + +There are two ways of visualizing recent build results: + +* The Web interface for each branch at http://python.org/dev/buildbot/, + where the so-called "waterfall" view presents a vertical rundown of recent + builds for each builder. When interested in one build, you'll have to + click on it to know which changesets it corresponds to. + +* The command-line ``bbreport.py`` client, which you can get from + http://code.google.com/p/bbreport/. Installing it is trivial: just add + the directory containing ``bbreport.py`` to your system path so that + you can run it from any filesystem location. For example, if you want + to display the latest build results on the development ("default") branch, + type:: + + bbreport.py -q 3.x + +Some buildbots are much faster than others. Over time, you will learn which +ones produce the quickest results after a build, and which ones take the +longest time. + +Also, when several changesets are pushed in a quick succession in the same +branch, it often happens that a single build is scheduled for all these +changesets. + +Stability +--------- + +A subset of the buildbots are marked "stable". They are taken into account +when making a new release. The rule is that no stable buildbot must witness +persistent failures when the release is cut. It is absolutely **vital** +that core developers fix any issue they introduce on the stable buildbots, +as soon as possible. + +This does not mean that other builders' test results can be taken lightly, +either. Some of them are known for having platform-specific issues that +prevent some tests from succeeding (or even terminating at all), but +introducing additional failures should generally not be an option. + +Flags-dependent failures +------------------------ + +Sometimes, while you have run the :doc:`whole test suite <runtests>` before +committing, you may witness unexpected failures on the buildbots. One source +of such discrepancies is if different flags have been passed to the test runner +or to Python itself. To reproduce, make sure you use the same flags as the +buildbots: they can be found out simply by clicking the **stdio** link for +the failing build's tests. For example:: + + ./python.exe -Wd -E -bb ./Lib/test/regrtest.py -uall -rwW + +.. note:: + Running ``Lib/test/regrtest.py`` is exactly equivalent to running + ``-m test``. + +Ordering-dependent failures +--------------------------- + +Sometimes even the failure is subtler, as it relies on the order in which +the tests are run. The buildbots *randomize* test order (by using the ``-r`` +option to the test runner) to maximize the probability that potential +interferences between library modules are exercised; the downside is that it +can make for seemingly sporadic failures. + +The ``--randseed`` option makes it easy to reproduce the exact randomization +used in a given build. Again, open the ``stdio`` link for the failing test +run, and check the beginning of the test output proper. + +Let's assume, for the sake of example, that the output starts with:: + + ./python -Wd -E -bb Lib/test/regrtest.py -uall -rwW + == CPython 3.3a0 (default:22ae2b002865, Mar 30 2011, 13:58:40) [GCC 4.4.5] + == Linux-2.6.36-gentoo-r5-x86_64-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_4400+-with-gentoo-1.12.14 little-endian + == /home/buildbot/buildarea/3.x.ochtman-gentoo-amd64/build/build/test_python_29628 + Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0) + Using random seed 2613169 + [ 1/353] test_augassign + [ 2/353] test_functools + +You can reproduce the exact same order using:: + + ./python -Wd -E -bb -m test -uall -rwW --randseed 2613169 + +It will run the following sequence (trimmed for brevity):: + + [ 1/353] test_augassign + [ 2/353] test_functools + [ 3/353] test_bool + [ 4/353] test_contains + [ 5/353] test_compileall + [ 6/353] test_unicode + +If this is enough to reproduce the failure on your setup, you can then +bisect the test sequence to look for the specific interference causing the +failure. Copy and paste the test sequence in a text file, then use the +``--fromfile`` (or ``-f``) option of the test runner to run the exact +sequence recorded in that text file:: + + ./python -Wd -E -bb -m test -uall -rwW --fromfile mytestsequence.txt + +In the example sequence above, if ``test_unicode`` had failed, you would +first test the following sequence:: + + [ 1/353] test_augassign + [ 2/353] test_functools + [ 3/353] test_bool + [ 6/353] test_unicode + +And, if it succeeds, the following one instead (which, hopefully, shall +fail):: + + [ 4/353] test_contains + [ 5/353] test_compileall + [ 6/353] test_unicode + +Then, recursively, narrow down the search until you get a single pair of +tests which triggers the failure. It is very rare that such an interference +involves more than **two** tests. If this is the case, we can only wish you +good luck! + +.. note:: + You cannot use the ``-j`` option (for parallel testing) when diagnosing + ordering-dependent failures. Using ``-j`` isolates each test in a + pristine subprocess and, therefore, prevents you from reproducing any + interference between tests. + + +Transient failures +------------------ + +While we try to make the test suite as reliable as possible, some tests do +not reach a perfect level of reproduceability. Some of them will sometimes +display spurious failures, depending on various conditions. Here are common +offenders: + +* Network-related tests, such as ``test_poplib``, ``test_urllibnet``, etc. + Their failures can stem from adverse network conditions, or imperfect + thread synchronization in the test code, which often has to run a + server in a separate thread. + +* Tests dealing with delicate issues such as inter-thread or inter-process + synchronization, or Unix signals: ``test_multiprocessing``, + ``test_threading``, ``test_subprocess``, ``test_threadsignals``. + +When you think a failure might be transient, it is recommended you confirm by +waiting for the next build. Still, even if the failure does turn out sporadic +and unpredictable, the issue should be reported on the bug tracker; even +better if it can be diagnosed and suppressed by fixing the test's implementation, +or by making its parameters - such as a timeout - more robust. + diff --git a/index.rst b/index.rst --- a/index.rst +++ b/index.rst @@ -21,6 +21,7 @@ developers committing devcycle + buildbots stdlibchanges langchanges @@ -40,7 +41,7 @@ contributing to Python. * `Issue tracker <http://bugs.python.org/>`_ -* Buildbots_ +* `Buildbot status`_ * :doc:`faq` * PEPs_ (Python Enhancement Proposals) @@ -69,7 +70,7 @@ * :doc:`coverage` * Advanced tasks for once you are comfortable * :doc:`silencewarnings` - * Fixing issues found by the buildbots_ + * Fixing issues found by the :doc:`buildbots <buildbots>` * :doc:`fixingissues` * :doc:`helptriage` * :doc:`devrole` @@ -79,6 +80,7 @@ * :doc:`coredev` * :doc:`committing` * :doc:`devcycle` + * :doc:`buildbots` Proposing changes to Python itself @@ -102,7 +104,7 @@ tracker for the issue tracker) * :doc:`experts` * `Firefox search engine plug-in`_ -* Buildbots_ +* `Buildbot status`_ * Source code * `Browse online <http://hg.python.org/cpython/>`_ * `Snapshot of py3k <http://hg.python.org/cpython/archive/tip.tar.bz2>`_ @@ -120,7 +122,7 @@ * :doc:`developers` -.. _buildbots: http://python.org/dev/buildbot/ +.. _Buildbot status: http://python.org/dev/buildbot/ .. _Firefox search engine plug-in: http://www.python.org/dev/searchplugin/ .. _Misc directory: http://hg.python.org/cpython/file/tip/Misc .. _PEPs: http://www.python.org/dev/peps -- Repository URL: http://hg.python.org/devguide