[Python-Dev] 2nd draft of "How Py is Developed" essay
bac at OCF.Berkeley.EDU
Fri Oct 24 17:53:56 EDT 2003
OK, so using the feedback from the first draft I made a few changes.
One is a paragraph on what to do if you want to add or change a file on
a patch item if you are not the original submitter. I also added
two-sentence conclusion to the whole essay. Lastly I changed the title
to better reflect how Python is ultimately developed. =)
As before, any comments and corrections are welcome. If you think this
sucker is done, please say so! If I get enough people saying they think
this is good enough to go out to the world I will post to
python-announce and python-list and then add it to the python.org/dev/ .
Then you can all hear me discuss it again at PyCon (assuming it gets
Guido, Some Guys, and a Mailing List: How Python is Developed
Software does not make itself. Code does not spontaneously come from
the ether of the universe. Python_ is no exception to this rule. Since
Python made its public debut back in 1991 people beyond the BDFL
(Benevolent Dictator For Life, `Guido van Rossum`_) have helped
contribute time and energy to making Python what it is today; a
powerful, simple programming language available to all.
But it has not been a random process of people doing whatever they
wanted to Python. Over the years a process to the development of Python
has emerged by the group that heads Python's growth and maintenance;
`python-dev`_. This document is an attempt to write this process down
in hopes of lowering any barriers possibly preventing people from
contributing to the development of Python.
.. _Python: http://www.python.org/
.. _Guido van Rossum: http://www.python.org/~guido/
To help facilitate the development of Python, certain tools are used.
Beyond the obvious ones such as a text editor and email client, two
tools are very pervasive in the development process.
SourceForge_ is used by python-dev to keep track of feature requests,
reported bugs, and contributed patches. A detailed explanation on how
to use SourceForge is covered later in `General SourceForge Guidelines`_.
CVS_ is a networked file versioning system that stores all of files that
make up Python. It allows the developers to have a single repository
for the files along with being able to keep track of any and all changes
to every file. The basic commands and uses can be found in the `dev
FAQ`_ along with a multitude of tutorials spread across the web.
.. _SourceForge: http://sourceforge.net/projects/python/
.. _CVS: http://www.cvshome.org/
.. _dev FAQ: http://www.python.org/dev/devfaq.html
Python development is not just programming. It requires a great deal of
communication between people. This communication is not just between
the members of python-dev; communication within the greater Python
community also helps with development. Several mailing lists and
newsgroups are used to help organize all of these discussions.
In terms of Python development, the primary location for communication
is the `python-dev`_ mailing list. This is where the members of
python-dev hash out ideas and iron out issues. It is an open list;
anyone can subscribe to the mailing list. While the discussion can get
quite technical, it is not all out of the reach for even a novice and
thus should not discourage anyone from joining the list. Please
realize, though, this list is **only** for the discussion of the
development of Python; all other questions should be directed somewhere
else, such as `python-list`_.
When the greater Python community is involved in a discussion, it always
ends up on `python-list`_. This mailing list is a gateway to the
newsgroup `comp.lang.python`_. This is also a good place to go when you
have a question about Python that does not pertain to the actual
development of the language.
Using CVS_ allows the development team to know who made a change to a
file and when they made their change. But unless one wants to
continuously update their local checkout of the repository, the best way
to stay on top of changes to the repository is to subscribe to
`Python-checkins`_. This list sends out an email for each and every
change to a file in Python. This list can generate a large amount of
traffic since even changing a typo in some text will trigger an email to
be sent out. But if you wish to be kept abreast of all changes to
Python then this is a good way to do so.
The Patches_ mailing list sends out an email for all changes to patch
items on SourceForge_. This list, just like Python-checkins, can
generate a large amount of email traffic. It is in general useful to
people who wish to help out with the development of Python by knowing
about all new submitted patches as well as any new developments on
`Python-bugs-list`_ functions much like the Patches mailing list except
it is for bug items on SourceForge. If you find yourself wanting to
help to close and remove bugs in Python this is the right list to
subscribe to if you can handle the volume of email.
.. _python-list: http://mail.python.org/mailman/listinfo/python-list
.. _comp.lang.python: http://groups.google.com/groups?q=comp.lang.python
.. _Python-checkins: http://mail.python.org/mailman/listinfo/python-checkins
.. _Patches: http://mail.python.org/mailman/listinfo/patches
The Actual Development
Developing Python is not all just conversations about neat new language
features (although those neat conversations do come up and there is a
process to it). Developing Python also involves maintaining it by
eliminating discovered bugs, adding and changing features, and various
other jobs that are not necessarily glamorous but are just as important
to the language as anything else.
General SourceForge Guidelines
Since a good amount of Python development involves using SourceForge_,
it is important to follow some guidelines when handling a tracker item
(bug, patch, etc.). Probably one of the most important things you can
do is make sure to set the various options in a new tracker item
properly. The submitter should make sure that the Data Type, Category,
and Group are all set to reasonable values. The remaining values
(Assigned To, Status, and Resolution) should in general be left to
Python developers to set. The exception to this rule is when you want
to retract a patch; then "close" the patch by setting Status to "closed"
and Resolution to whatever is appropriate.
Make sure you do a cursory check to make sure what ever you are
submitting was not previously submitted by someone else. Duplication
just uses up valuable time.
And **please** do not post feature requests, bug reports, or patches to
the python-dev mailing list. If you do you will be instructed to create
an appropriate SourceForge tracker item. When in doubt as to whether
you should bring something to python-dev's attention, you can always ask
on `comp.lang.python`_; Python developers actively participate there and
move the conversation over if it is deemed reasonable.
`Feature requests`_ are for features that you wish Python had but you
have no plans on actually implementing by writing a patch. On occasion
people do go through the features requests (also called RFEs on
SourceForge) to see if there is anything there that they think should be
implemented and actually do the implementation. But in general do not
expect something put here to be implemented without some participation
on your part.
The best way to get something implemented is to campaign for it in the
greater Python community. `comp.lang.python`_ is the best place to
accomplish this. Post to the newsgroup with your idea and see if you
can either get support or convince someone to implement it. It might
even end up being added to `PEP 42`_ so that the idea does not get lost
in the noise as time passes.
.. _feature requests:
.. _PEP 42: http://www.python.org/peps/pep-0042.html
Think you found a bug? Then submit a `bug report`_ on SourceForge.
Make sure you clearly specify what version of Python you are using, what
OS, and under what conditions the bug was triggered. The more
information you can give the faster the bug can be fixed since time will
not be wasted requesting more information from you.
.. _bug report: http://sourceforge.net/tracker/?group_id=5470&atid=105470
Create a patch_ tracker item on SourceForge for any code you think
should be applied to the Python CVS tree. For practically any change to
Python's functionality the documentation and testing suite will need to
be changed as well. Doing this in the first place speeds things up
Please make sure your patch is against the CVS repository. If you don't
know how to use it (basics are covered in the `dev FAQ`_), then make
sure you specify what version of Python you made your patch against.
In terms of coding standards, `PEP 8`_ specifies for Python while `PEP
7`_ specifies for C. Always try to maximize your code reuse; it makes
maintenance much easier.
For C code make sure to limit yourself to ANSI C code as much as
possible. If you must use non-ANSI C code then see if what you need is
checked for by looking in pyconfig.h . You can also look in
Include/pyport.h for more helpful C code. If what you need is still not
there but it is in general available, then add a check in configure.in
for it (don't forget to run autoreconf to make the changes to take
effect). And if that *still* doesn't fit your needs then code up a
solution yourself. The reason for all of this is to limit the
dependence on external code that might not be available for all OSs that
Python runs on.
Be aware of intellectual property when handling patches. Any code with
no copyright will fall under the copyright of the `Python Software
Foundation`_. If you have no qualms with that, wonderful; this is the
best solution for Python. But if you feel the need to include a
copyright then make sure that it is compatible with copyright used on
Python (i.e., BSD-style). The best solution, though, is to sign the
copyright over to the Python Software Foundation.
.. _patch: http://sourceforge.net/tracker/?group_id=5470&atid=305470
.. _dev FAQ: http://www.python.org/dev/devfaq.html
.. _PEP 7: http://www.python.org/peps/pep-0007.html
.. _PEP 8: http://www.python.org/peps/pep-0008.html
.. _Python Software Foundation: http://www.python.org/psf/
Changing the Language
You understand how to file a patch. You think you have a great idea on
how Python should change. You are ready to write code for your change.
Great, but you need to realize that certain things must be done for a
change to be accepted. Changes fall into two categories; changes to the
standard library (referred to as the "stdlib") and changes to the
Changes to the stdlib
Changes to the stdlib can consist of adding functionality or changing
Adding minor functionality (such as a new function or method) requires
convincing a member of python-dev that the addition of code caused by
implementing the feature is worth it. A big addition such as a module
tends to require more support than just a single member of python-dev.
As always, getting community support for your addition is a good idea.
With all additions, make sure to write up documentation for your new
functionality. Also make sure that proper tests are added to the
If you want to add a module, be prepared to be called upon for any bug
fixes or feature requests for that module. Getting a module added to
the stdlib makes you by default its maintainer. If you can't take that
level of responsibility and commitment and cannot get someone else to
take it on for you then your battle will be very difficult; when there
is not a specific maintainer of code python-dev takes responsibility and
thus your code must be useful to them or else they will reject the module.
Changing existing functionality can be difficult to do if it breaks
backwards-compatibility. If your code will break existing code, you
must provide a legitimate reason on why making the code act in a
non-compatible way is better than the status quo. This requires
python-dev as a whole to agree to the change.
Changing the Language Proper
Changing Python the language is taken **very** seriously. Python is
often heralded for its simplicity and cleanliness. Any additions to the
language must continue this tradition and view. Thus any changes must
go through a long process.
First, you must write a PEP_ (Python Enhancement Proposal). This is
basically just a document that explains what you want, why you want it,
what could be bad about the change, and how you plan on implementing the
change. It is best to get feedback on PEPs on `comp.lang.python`_ and
from python-dev. Once you feel the document is ready you can request a
PEP number and to have it added to the official list of PEPs in `PEP 0`_.
Once you have a PEP, you must then convince python-dev and the BDFL that
your change is worth it. Be expected to be bombarded with questions and
counter-arguments. It can drag on for over a month, easy. If you are
not up for that level of discussion then do not bother with trying to
get your change in. If you manage to convince a majority of python-dev
and the BDFL (or most of python-dev; that can lead to the BDFL changing
his mind) then your change can be applied.
As with all new code make sure you also have appropriate documentation
patches along with tests for the new functionality.
.. _PEP: http://www.python.org/peps/pep-0001.html
.. _PEP 0: http://www.python.org/peps/pep-0000.html
Many people say they wish they could help out with the development of
Python but feel they are not up to writing code. There are plenty of
things one can do, though, that does not require you to write code.
Regardless of your coding abilities, there is something for everyone to
For feature requests, adding a comment about what you think is helpful.
State whether or not you would like to see the feature. You can also
volunteer to write the code to implement the feature if you feel up to it.
For bugs, stating whether or not you can reproduce the bug yourself can
be extremely helpful. If you can write a fix for the bug that is very
helpful as well; start a patch item and reference it in a comment in the
For patches, apply the patch and run the testing suite. You can do a
code review on the patch to make sure that it is good, clean code. If
the patch adds a new feature, comment on whether you think it is worth
adding. If it changes functionality then comment on whether you think
it might break code; if it does, say whether you think it is worth the
cost of breaking existing code. Help add to the patch if it is missing
documentation patches or needed regression tests.
A special mention about adding a file to a tracker item. Only official
developers and the creator of the tracker item can add a file. This
means that if you want to add a file and you are neither of the types of
people just mentioned you have to do an extra step or two. One thing
you can do is post the file you want added somewhere else online and
reference the URL in a comment. You can also create a new patch item if
you feel the change is thorough enough and cross-reference between both
patches in the comments. Be wary of this last option, though, since
some people might be offended since it might come off as if you think
there code is bad and yours is better. The best solution of all is to
work with the original poster if they are receptive to help. But if
they do not respond or are not friendly then do go ahead and do one of
the other two suggestions.
For language changes, make your voice be heard. Comment about any PEPs
on `comp.lang.python`_ so that the general opinion of the community can
If there is nothing specific you find you want to work on but still feel
like contributing nonetheless, there are several things you can do. The
documentation can always use fleshing out. Adding more tests to the
testing suite is always useful. Contribute to discussions on python-dev
or `comp.lang.python`_. Just helping out in the community by spreading
the word about Python or helping someone with a question is helpful.
If you really want to get knee-deep in all of this, join python-dev.
Once you have been actively participating for a while and are generally
known on python-dev you can request to have checkin rights on the CVS
tree. It is a great way to learn how to work in a large, distributed
group along with how to write great code.
And if all else fails give money; the `Python Software Foundation`_ is a
non-profit organization that accepts donations that are tax-deductible
in the United States. The funds are used for various thing such as
lawyers for handling the intellectual property of Python to funding
PyCon_. But the PSF could do a lot more if they had the funds. One
goal is to have enough money to fund having Guido work on Python for a
full year full-time; this would bring about Python 3. Every dollar does
help, so please contribute if you can.
.. _PyCon: http://www.python.org/pycon/
If you get any message from this document, it should be that *anyone*
can help with the development of Python. All help is greatly
appreciated and keeps the language the wonderful piece of software that
More information about the Python-Dev