[Python-Dev] 2nd draft of "How Py is Developed" essay

[Brett C.]

OK, so using the feedback from the first draft I made a few changes. 
One is a paragraph on what to do if you want to add or change a file on 
a patch item if you are not the original submitter.  I also added 
two-sentence conclusion to the whole essay.  Lastly I changed the title 
to better reflect how Python is ultimately developed.  =)

As before, any comments and corrections are welcome.  If you think this 
sucker is done, please say so!  If I get enough people saying they think 
this is good enough to go out to the world I will post to 
python-announce and python-list and then add it to the python.org/dev/ . 
  Then you can all hear me discuss it again at PyCon (assuming it gets 
accepted).  =)

----------------------------

Guido, Some Guys, and a Mailing List: How Python is Developed
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Introduction
============
Software does not make itself.  Code does not spontaneously come from 
the ether of the universe.  Python_ is no exception to this rule.  Since 
Python made its public debut back in 1991 people beyond the BDFL 
(Benevolent Dictator For Life, `Guido van Rossum`_) have helped 
contribute time and energy to making Python what it is today; a 
powerful, simple programming language available to all.

But it has not been a random process of people doing whatever they 
wanted to Python.  Over the years a process to the development of Python 
has emerged by the group that heads Python's growth and maintenance; 
`python-dev`_.  This document is an attempt to write this process down 
in hopes of lowering any barriers possibly preventing people from 
contributing to the development of Python.

.. _Python: http://www.python.org/
.. _Guido van Rossum: http://www.python.org/~guido/
.. _python-dev:http://mail.python.org/mailman/listinfo/python-dev


Tools Used
==========
To help facilitate the development of Python, certain tools are used. 
Beyond the obvious ones such as a text editor and email client, two 
tools are very pervasive in the development process.

SourceForge_ is used by python-dev to keep track of feature requests, 
reported bugs, and contributed patches.  A detailed explanation on how 
to use SourceForge is covered later in `General SourceForge Guidelines`_.

CVS_ is a networked file versioning system that stores all of files that 
make up Python.  It allows the developers to have a single repository 
for the files along with being able to keep track of any and all changes 
to every file.  The basic commands and uses can be found in the `dev 
FAQ`_ along with a multitude of tutorials spread across the web.

.. _SourceForge: http://sourceforge.net/projects/python/
.. _CVS: http://www.cvshome.org/
.. _dev FAQ: http://www.python.org/dev/devfaq.html


Communicating
=============
Python development is not just programming.  It requires a great deal of 
communication between people.  This communication is not just between 
the members of python-dev; communication within the greater Python 
community also helps with development.  Several mailing lists and 
newsgroups are used to help organize all of these discussions.

In terms of Python development, the primary location for communication 
is the `python-dev`_ mailing list.  This is where the members of 
python-dev hash out ideas and iron out issues.  It is an open list; 
anyone can subscribe to the mailing list.  While the discussion can get 
quite technical, it is not all out of the reach for even a novice and 
thus should not discourage anyone from joining the list.  Please 
realize, though, this list is **only** for the discussion of the 
development of Python; all other questions should be directed somewhere 
else, such as `python-list`_.

When the greater Python community is involved in a discussion, it always 
ends up on `python-list`_.  This mailing list is a gateway to the 
newsgroup `comp.lang.python`_.  This is also a good place to go when you 
have a question about Python that does not pertain to the actual 
development of the language.

Using CVS_ allows the development team to know who made a change to a 
file and when they made their change.  But unless one wants to 
continuously update their local checkout of the repository, the best way 
to stay on top of changes to the repository is to subscribe to 
`Python-checkins`_.  This list sends out an email for each and every 
change to a file in Python.  This list can generate a large amount of 
traffic since even changing a typo in some text will trigger an email to 
be sent out.  But if you wish to be kept abreast of all changes to 
Python then this is a good way to do so.

The Patches_ mailing list sends out an email for all changes to patch 
items on SourceForge_.  This list, just like Python-checkins, can 
generate a large amount of email traffic.  It is in general useful to 
people who wish to help out with the development of Python by knowing 
about all new submitted patches as well as any new developments on 
preexisting ones.

`Python-bugs-list`_ functions much like the Patches mailing list except 
it is for bug items on SourceForge.  If you find yourself wanting to 
help to close and remove bugs in Python this is the right list to 
subscribe to if you can handle the volume of email.

.. _python-list: http://mail.python.org/mailman/listinfo/python-list
.. _comp.lang.python: http://groups.google.com/groups?q=comp.lang.python
.. _Python-checkins: http://mail.python.org/mailman/listinfo/python-checkins
.. _Patches: http://mail.python.org/mailman/listinfo/patches
.. _Python-bugs-list: 
http://mail.python.org/mailman/listinfo/python-bugs-list


The Actual Development
======================
Developing Python is not all just conversations about neat new language 
features (although those neat conversations do come up and there is a 
process to it).  Developing Python also involves maintaining it by 
eliminating discovered bugs, adding and changing features, and various 
other jobs that are not necessarily glamorous but are just as important 
to the language as anything else.


General SourceForge Guidelines
------------------------------
Since a good amount of Python development involves using SourceForge_, 
it is important to follow some guidelines when handling a tracker item 
(bug, patch, etc.).  Probably one of the most important things you can 
do is make sure to set the various options in a new tracker item 
properly.  The submitter should make sure that the Data Type, Category, 
and Group are all set to reasonable values.  The remaining values 
(Assigned To, Status, and Resolution) should in general be left to 
Python developers to set.  The exception to this rule is when you want 
to retract a patch; then "close" the patch by setting Status to "closed" 
and Resolution to whatever is appropriate.

Make sure you do a cursory check to make sure what ever you are 
submitting was not previously submitted by someone else.  Duplication 
just uses up valuable time.

And **please** do not post feature requests, bug reports, or patches to 
the python-dev mailing list.  If you do you will be instructed to create 
an appropriate SourceForge tracker item.  When in doubt as to whether 
you should bring something to python-dev's attention, you can always ask 
on `comp.lang.python`_; Python developers actively participate there and 
move the conversation over if it is deemed reasonable.


Feature Requests
----------------
`Feature requests`_ are for features that you wish Python had but you 
have no plans on actually implementing by writing a patch.  On occasion 
people do go through the features requests (also called RFEs on 
SourceForge) to see if there is anything there that they think should be 
implemented and actually do the implementation.  But in general do not 
expect something put here to be implemented without some participation 
on your part.

The best way to get something implemented is to campaign for it in the 
greater Python community.  `comp.lang.python`_ is the best place to 
accomplish this.  Post to the newsgroup with your idea and see if you 
can either get support or convince someone to implement it.  It might 
even end up being added to `PEP 42`_ so that the idea does not get lost 
in the noise as time passes.

.. _feature requests: 
http://sourceforge.net/tracker/?group_id=5470&atid=355470
.. _PEP 42: http://www.python.org/peps/pep-0042.html


Bug Reports
-----------
Think you found a bug?  Then submit a `bug report`_ on SourceForge. 
Make sure you clearly specify what version of Python you are using, what 
OS, and under what conditions the bug was triggered.  The more 
information you can give the faster the bug can be fixed since time will 
not be wasted requesting more information from you.

.. _bug report: http://sourceforge.net/tracker/?group_id=5470&atid=105470


Patches
-------
Create a patch_ tracker item on SourceForge for any code you think 
should be applied to the Python CVS tree.  For practically any change to 
Python's functionality the documentation and testing suite will need to 
be changed as well.  Doing this in the first place speeds things up 
considerably.

Please make sure your patch is against the CVS repository.  If you don't 
know how to use it (basics are covered in the `dev FAQ`_), then make 
sure you specify what version of Python you made your patch against.

In terms of coding standards, `PEP 8`_ specifies for Python while `PEP 
7`_ specifies for C.  Always try to maximize your code reuse; it makes 
maintenance much easier.

For C code make sure to limit yourself to ANSI C code as much as 
possible.  If you must use non-ANSI C code then see if what you need is 
checked for by looking in pyconfig.h .  You can also look in 
Include/pyport.h for more helpful C code.  If what you need is still not 
there but it is in general available, then add a check in configure.in 
for it (don't forget to run autoreconf to make the changes to take 
effect).  And if that *still* doesn't fit your needs then code up a 
solution yourself.  The reason for all of this is to limit the 
dependence on external code that might not be available for all OSs that 
Python runs on.

Be aware of intellectual property when handling patches.  Any code with 
no copyright will fall under the copyright of the `Python Software 
Foundation`_.  If you have no qualms with that, wonderful; this is the 
best solution for Python.  But if you feel the need to include a 
copyright then make sure that it is compatible with copyright used on 
Python (i.e., BSD-style).  The best solution, though, is to sign the 
copyright over to the Python Software Foundation.

.. _patch: http://sourceforge.net/tracker/?group_id=5470&atid=305470
.. _dev FAQ: http://www.python.org/dev/devfaq.html
.. _PEP 7: http://www.python.org/peps/pep-0007.html
.. _PEP 8: http://www.python.org/peps/pep-0008.html
.. _Python Software Foundation: http://www.python.org/psf/


Changing the Language
=====================
You understand how to file a patch.  You think you have a great idea on 
how Python should change.  You are ready to write code for your change. 
  Great, but you need to realize that certain things must be done for a 
change to be accepted.  Changes fall into two categories; changes to the 
standard library (referred to as the "stdlib") and changes to the 
language proper.


Changes to the stdlib
---------------------
Changes to the stdlib can consist of adding functionality or changing 
existing functionality.

Adding minor functionality (such as a new function or method) requires 
convincing a member of python-dev that the addition of code caused by 
implementing the feature is worth it.  A big addition such as a module 
tends to require more support than just a single member of python-dev. 
As always, getting community support for your addition is a good idea.

With all additions, make sure to write up documentation for your new 
functionality.  Also make sure that proper tests are added to the 
testing suite.

If you want to add a module, be prepared to be called upon for any bug 
fixes or feature requests for that module.  Getting a module added to 
the stdlib makes you by default its maintainer.  If you can't take that 
level of responsibility and commitment and cannot get someone else to 
take it on for you then your battle will be very difficult; when there 
is not a specific maintainer of code python-dev takes responsibility and 
thus your code must be useful to them or else they will reject the module.

Changing existing functionality can be difficult to do if it breaks 
backwards-compatibility.  If your code will break existing code, you 
must provide a legitimate reason on why making the code act in a 
non-compatible way is better than the status quo.  This requires 
python-dev as a whole to agree to the change.

Changing the Language Proper
----------------------------
Changing Python the language is taken **very** seriously.  Python is 
often heralded for its simplicity and cleanliness.  Any additions to the 
language must continue this tradition and view.  Thus any changes must 
go through a long process.

First, you must write a PEP_ (Python Enhancement Proposal).  This is 
basically just a document that explains what you want, why you want it, 
what could be bad about the change, and how you plan on implementing the 
change.  It is best to get feedback on PEPs on `comp.lang.python`_ and 
from python-dev.  Once you feel the document is ready you can request a 
PEP number and to have it added to the official list of PEPs in `PEP 0`_.

Once you have a PEP, you must then convince python-dev and the BDFL that 
your change is worth it.  Be expected to be bombarded with questions and 
counter-arguments.  It can drag on for over a month, easy.  If you are 
not up for that level of discussion then do not bother with trying to 
get your change in.  If you manage to convince a majority of python-dev 
and the BDFL (or most of python-dev; that can lead to the BDFL changing 
his mind) then your change can be applied.

As with all new code make sure you also have appropriate documentation 
patches along with tests for the new functionality.

.. _PEP: http://www.python.org/peps/pep-0001.html
.. _PEP 0: http://www.python.org/peps/pep-0000.html


Helping Out
===========
Many people say they wish they could help out with the development of 
Python but feel they are not up to writing code.  There are plenty of 
things one can do, though, that does not require you to write code. 
Regardless of your coding abilities, there is something for everyone to 
help with.

For feature requests, adding a comment about what you think is helpful. 
  State whether or not you would like to see the feature.  You can also 
volunteer to write the code to implement the feature if you feel up to it.

For bugs, stating whether or not you can reproduce the bug yourself can 
be extremely helpful.  If you can write a fix for the bug that is very 
helpful as well; start a patch item and reference it in a comment in the 
bug item.

For patches, apply the patch and run the testing suite.  You can do a 
code review on the patch to make sure that it is good, clean code.  If 
the patch adds a new feature, comment on whether you think it is worth 
adding.  If it changes functionality then comment on whether you think 
it might break code; if it does, say whether you think it is worth the 
cost of breaking existing code.  Help add to the patch if it is missing 
documentation patches or needed regression tests.

A special mention about adding a file to a tracker item.  Only official 
developers and the creator of the tracker item can add a file.  This 
means that if you want to add a file and you are neither of the types of 
people just mentioned you have to do an extra step or two.  One thing 
you can do is post the file you want added somewhere else online and 
reference the URL in a comment.  You can also create a new patch item if 
you feel the change is thorough enough and cross-reference between both 
patches in the comments.  Be wary of this last option, though, since 
some people might be offended since it might come off as if you think 
there code is bad and yours is better.  The best solution of all is to 
work with the original poster if they are receptive to help.  But if 
they do not respond or are not friendly then do go ahead and do one of 
the other two suggestions.

For language changes, make your voice be heard.  Comment about any PEPs 
on `comp.lang.python`_ so that the general opinion of the community can 
be assessed.

If there is nothing specific you find you want to work on but still feel 
like contributing nonetheless, there are several things you can do.  The 
documentation can always use fleshing out.  Adding more tests to the 
testing suite is always useful.  Contribute to discussions on python-dev 
or `comp.lang.python`_.  Just helping out in the community by spreading 
the word about Python or helping someone with a question is helpful.

If you really want to get knee-deep in all of this, join python-dev. 
Once you have been actively participating for a while and are generally 
known on python-dev you can request to have checkin rights on the CVS 
tree.  It is a great way to learn how to work in a large, distributed 
group along with how to write great code.

And if all else fails give money; the `Python Software Foundation`_ is a 
non-profit organization that accepts donations that are tax-deductible 
in the United States.  The funds are used for various thing such as 
lawyers for handling the intellectual property of Python to funding 
PyCon_.  But the PSF could do a lot more if they had the funds.  One 
goal is to have enough money to fund having Guido work on Python for a 
full year full-time; this would bring about Python 3.  Every dollar does 
help, so please contribute if you can.

.. _PyCon: http://www.python.org/pycon/


Conclusion
==========
If you get any message from this document, it should be that *anyone* 
can help with the development of Python.  All help is greatly 
appreciated and keeps the language the wonderful piece of software that 
it is.