[pypy-svn] r20553 - pypy/extradoc/talk/22c3

lac at codespeak.net lac at codespeak.net
Thu Dec 1 23:52:58 CET 2005

Author: lac
Date: Thu Dec  1 23:52:57 2005
New Revision: 20553

>is my comments.

There is still a lot of clunky English, but this before the deadline ...

Added: pypy/extradoc/talk/22c3/lac.comments
--- (empty file)
+++ pypy/extradoc/talk/22c3/lac.comments	Thu Dec  1 23:52:57 2005
@@ -0,0 +1,367 @@
+Open Source and EU funding: agile methods in a funded OSS project
+This paper walks through different aspects of agility 
+within the open-source driven PyPy project.  
+>what's 'Open Source Driven'?
+played a key role 
+from the beginning - the PyPy project started from some mails
+between a few people, quickly had a first one-week meeting, a "sprint"
+where it evolved into a structure that was able to carry
+out a research project - and got accepted by the European
+Union.  During the course, two companies got founded and are
+now growing and employing key developers. 
+PyPy's technical development is strongly rooted in open-source
+contexts and this adds another agility aspect - free
+communication, co-operation and exchange with other people and
+The process of obtaining EU-funding is a continous challenge to 
+the community-rooted PyPy project: how to connect
+agile open source culture with formal structures: interaction
+with requirements like planning, budget estimation, work
+distribution and resource tracking.  
+After our first "funded" year we are reasonably happy with the
+balance we strike between organisations and EU funding on the
+one and the developers driving the technical aspects of the
+project on the other side. 
+Agility in Technical Development and Organisation 
+Agile approaches: sprinting
+>PyPy first  (with bits I am seeing something dismembered -- bloody bits)
+started during a one-week meeting, a "sprint",
+held at Trillke-Gut in Hildesheim February 2003. The sprint was inspired by 
+practices used by other Python oriented projects such as Zope3.  Originally the 
+sprint methodology used in the Python community grew from practices applied by
+the Zope Corporation.  Their definition of a sprint was  "two-day or three-day focused
+development session, in which developers pair off together in a room and focus
+on building a particular subsystem". 
+Sprinting up to a week became the initial driving factor in developing 
+the code base and the community/people around it.  The early PyPy sprints 
+were organised by core developers together with local Pythonistas 
+in Louvain La Neuve, Gothenburg, Vilnius and Amsterdam.  Sprints gave
+the opportunity to both help, participate and influence the ideas within PyPy.
+Sprints are actually not part of the traditional Agile
+portfolio of techniques, the closest thing to it comes from
+Scrum who names the 30 days long programming iterations
+"sprints", covering a certain increment.  With the Scrum method, 
+considerable effort is put into performing the sprint planning
+as well as creating and documenting the "sprint backlog" which
+is then feedbacked into the "Product backlog".  The sprint ends
+with a "sprint review" - an informal planning session in which
+the team decides on upcoming work.  There are also techniques
+in which the team looks at ways to improve the development
+methodology and future sprints.
+To our knowledge, open-source projects these days are sprinting for at most
+a week which reflects the fact that many contributors give their time 
+and even money to gather and work together.  This is different 
+from having fully funded people from one company working together. 
+Why did PyPy choose sprinting as a key technique?  It is a method that fits
+distributed teams well because it gets the team focused around visible 
+challenging goals while working collarobatively (pair-programming, status
+meetings, discussions etc) as well as acceleratedly (short increments and
+tasks, "doing" and testing instead of long startups of planning and
+requirement gathering).   This means that most of the time a sprint is a great
+way of getting results and getting new people aquainted - a good 
+method for dissemination of knowledge and learning within the team. 
+.. image:: sprintprocess.gif 
+>Interesting.  I think that you have missed what I think is the most
+>important aspect.  Most EU projects do very poorly.  They define
+>success as 'whatever they did' so that it is rare for an EU project
+>to ever formally fail -- but people with a less charitable
+>interpretation of failure might announce that they fail a lot.
+>I think it is because the whole notion of 'divide the job into tasks,
+>and let different groups work on a different piece and then try to
+>stitch it all together through meetings' is a very flawed way to work.
+>This is creative work we are doing, not assembly line production.
+>If you want creative people to actually produce creative work, then you
+>have to let them work together.
+Agile approaches: test-driven development
+Test-driven development is a technical cornerstone for programming
+efficiently together in a distributed team.  Seen from the
+Agile Manifesto perspective it is right up there as one of the
+key elements since it puts focus on producing working code,
+rather than diagrams, plans and papers (and then faulty
+Seen from an Open Source community perspective it is a vitalising strategy -
+especially in combination with a transparent open process in which anyone
+interested can participate - if only for just a few days at a sprint.  Some of
+the key problems identified by Frederick P. Brooks in the latest version of
+"The Mythical Man-Month" (unfortunately still very actual today) are estimating
+correct amount of time for communication and testing/debugging.  Automated
+testing, rather barrier-free communication and strict version tracking helps 
+with those problems, especially in the hands of a team sprinting its way through 
+the Python community - welcoming everyone to participate.
+Apart from rewriting a practical programming language within
+itself, PyPy also evolved a number of development tools useful
+for writing tests and glueing things together. PyPy's testing
+tool ("py.test") is used separately and evolves on its own by now. 
+.. image:: plots/loc.png 
+.. test driven development 
+Agility: Open Communication and organisation
+Another agility aspect relates to transparent and open
+communication within the project.  Only very few (EU-contract
+related) documents are access restricted, everything else is
+freely available.  There are no hierarchies for commit rights.
+In fact, the hosting server also gives home to a couple of
+other projects and all projects share commit rights ("Coding
+Wiki").  Announcing Sprints, Releases and development goals
+lead to increasdingly many people subscribing to mailing lists or
+participating in development. 
+Moreover, the PyPy developers installed a model of weekly 30-minute 
+IRC chat meetings where topics are briefly discussed, delegated
+or decided upon: those meetings are open to all active developers 
+and usually do not discuss internal EU matters much except that 
+funded developers keep EU goals more in mind than others. 
+Minutes of these weekly developer meetings get archived and posted 
+to the development list. 
+A rather recent invention is the postings of "This week in PyPy" 
+which try to summarize what is going on in the lively IRC 
+development #pypy channel - main place of technical coordination. 
+.. image:: plots/subscribers.png 
+.. overview of PyPy mailing list subscriptions 
+How and why EU funding? 
+Mid 2003 the idea of trying to get EU-funding for the project was born. 
+It became clear that the project had an arbitrarily large scale and that
+receiving some funding would dramatically increase the pace and seriousness
+of the project - because funded developers can dedicate more of their time
+to the project.  The involved developers and people stretched outside of the 
+Open Source ecologies to try to gather as much information and contacts as 
+possible in order to answer the question: "Should we go for it?"  to which 
+the answer quickly became "Let's see how far we get!". 
+Making things fit with EU perspectives 
+There had been a growing interest from the European Commission, IST division,
+to look closer at the Open Source world and its achievements. Several funded
+research projects in the 5th framework programme studied the phenomenon
+(FLOSS-POLS, FLOSS) - its organization, business models and licensings. A few
+other funded software projects used Open Source in their work as tools
+(languages and applications).  There was no previous experience of an Open
+Source community based project making a bid for funding. 
+The areas in the 6th Framework programme (second call) fitted very well 
+with the objectives of PyPy.  The idea of strengthening the European Software
+development companies and businesses with supporting an open source language
+implementation was new but appealing to the EU.  But being an Open Source
+project wasn´t enough - the challenges and the idea of an flexible,
+configurable "translator" or "compiler" met the research targets of the FP6, as
+well as trying out and documenting the agile methodology being used.  
+It is interesting to note that 
+> most of today's computer language research and development happens in the US. 
+In short, we argued that EU funding allows the project to go for
+reaching a critical mass and position to continue to evolve from 
+there and that it would help European Organisations to make some
+Acting on this strategy proved to be a more difficult task. The
+entire proposal and negotiation process took over a year (Autumn 2003 till 
+November 2004). 
+>a proper description of planned work, 
+>necessary to satisfy formal requirements
+had not previously been part of the development focus and both the EU 
+and the parties involved had to adapt to the situation.  Yet, drafting the
+high-level requirements (in total 14 workpackages and 58 deliverables) was made
+using the same version-control/open-communication based work style, including
+evolving the proposal at sprints.  Writing the proposal and specifying according 
+objectives on a higher level has proved to be generally useful for clarifying goals 
+on a longer term - also helping others better understand the project. 
+Unfortunately the negotiations with the EU got stuck in
+organizational limbo and the project is still suffering from
+the effects of this even today.  The goal of funding
+contributors especially coming to sprints was originally
+based on a non-profit association.  This solution
+wasn't seen as realistic or feasible by the EU although
+we think it remains a viable approach for the future.   During
+negotiations, we got to an alternative solution which - however - 
+has a few drawbacks:  Contributors have to become Contract Partners within 
+the EU-level Consortium (which is by itself not hard) and can then at least
+claim travel and accomodation costs when attending sprints.
+However, this construction does not allow them to get paid for
+work time and also has some formal requirements.  This practically 
+leads to current considerations of developers to shift private money
+between them in order to circumvent the current problems with
+implementing an agile model within the EU contract framing. 
+Seven Organisations / The consortium 
+The guiding idea for receiving funding is to have organisations 
+through which key developers and other parties are employed. 
+Two companies out of the seven organisations in the initial
+consortium got funded during the EU negotiation process - 
+what first might have felt as an EU-related obstacle became an
+opportunity, but with some overhead in legal and organizational 
+Other adjustments and recruiting companies with previous EU
+project experiences took place.  There also is one company 
+involved quite unrelated to the previous developer work - 
+but rather focused on process management and
+designing learning processes with a background from the
+Chaospilot school in Aarhus, Denmark.  When creating the formal
+consortium of seven partners, new cultures and perspectives
+were mixed with the strong collaborative Open Source core
+team, adding new complexities in communication and
+cooperation.  Getting the new "playmates" to adopt the vision,
+culture and spirit of the original idea and holding true to it
+during the work on the proposal and negotiation process was a
+challenge indeed.
+The formal project organization required by the EU imposed
+more structure on the previous more free-floating agile
+process. Roles and responsibilities where staked out,
+conforming with the requirements of the roles but delegating
+as much as possible of the responsibilities and
+decision-making to the core developers.  The strategy was to
+keep "conceptual integrity" (Brooks) of the vision and the
+idea in the hands of the core developers.  A somewhat negative
+result was the added workload and responsibility on developers
+regarding EU related work.  It is not too surprising that
+the consortium with its member organisation now employs a
+version-control/review based scheme regarding EU documents
+reflecting the technical development approaches. 
+It remains a challenge for all partners of the consortium,
+universities and companies alike, to connect an ongoing
+medium-scale open-source project with EU regulations and
+requirements - not to speak of the fact that companies need to
+fund 50% of the costs themselves.  It is, in fact, too early
+to judge on the overall success of our approaches although 
+we are confident that things work out reasonably well. 
+challenge: balancing community interests with EU requirements
+The agile development process in the EU funded work of the
+PyPy project centers around sprints - which are planned to
+take place every 6th week at different places to allow many
+developers to get in direct touch with each other.  Sprinting
+around conferences also became a key strategy.
+But the nature of sprints changed when EU funding started.  The
+need to meet milestones of promised *deliverables* and the
+goal to keep an open sprint process, still welcoming newcomers
+into the world of Pypy, made the sprints longer (at least 7
+days with a break day in the middle) but also changed the
+nature of the sprints. The team started to distuingish between
+sprints open for all to attend, without any prior PyPy experience,
+and sprints requiring earlier PyPy involvement.  Tutorials, start up
+planning meetings as well as daily status meetings evolved,
+the latest additions to the sprints are closing planning
+meetings (planning the work between sprints) and work-groups -
+a version of pair-programming in groups.
+Some other effects of sprinting within the EU-structure is that the sprint
+becomes a forum for non-development work - coordinating and tracking the
+project. The challenge here is not affecting the main work and "disturbing"
+visiting developers with EU-related work. It could also be argued that the
+prolonged sprints could possibly make it more difficult for non consortium
+members to attend the full time, disturbing other engagements etc. 
+The project continues to try to enhance the method of sprinting, evaluating
+feedback from sprint participants.  Maybe the implementation within the PyPy
+team is slowly conforming to the Scrum standard of sprinting, but not as a
+conscious effort?
+Managing diversities: agile business - a succesful marriage?
+For a diverse group of organisations and people, agility is
+helpful at various levels: you cannot make all-encompassing
+plans and hope to statically follow them and succeed.  New
+developments, twists and opportunities evolve all the time.  
+Our experience with evolving PyPy from a loose Open Source project
+to a partially funded EU research project shows the following:
+- what first seemed like too diverse interests and views,
+  impossible to tailor into a single project, was instead a
+  fruitful mix of diversities. The challenge is to manage
+  these diversities and channel them into constructive team
+  efforts.  Aiming for homogenity is the real threat.  
+- what first seemed like unbeatable odds and too big obstacles
+  even turned into new possibilities. The challenge is
+  to maintain an atmosphere in which a team can act on those and
+  within short timeframes of opportunities.  Change is
+  inevitable - how you handle it is the real challenge.
+- there are many other projects and organisations who are 
+  heading in similar directions of trying to connect and
+  evolve agile open source strategies with business matters. 
+  Emerging models for developers distributed between
+  different countries allows people with special interests
+  to effectively work together and learn from each other. 
+Concluding  - the cumulative effects of an agile, open and
+dynamic team process combined with a market and curious
+first adopters facilitates agile business.  A positive result 
+is that a lot of people within the PyPy context found 
+enjoyable jobs and there now already is evolving commercial
+interest despite the still early stages of the project - mostly 
+from US companies though ... why european companies, especially
+larger ones, appear to prefer taking rather naive views on agile 
+open-source development ("great, it's cheaper, no license fees!") 
+is another interesting topic. 

More information about the Pypy-commit mailing list