[pypy-commit] extradoc extradoc: merge

cfbolz noreply at buildbot.pypy.org
Tue Jan 3 14:30:37 CET 2012

Author: Carl Friedrich Bolz <cfbolz at gmx.de>
Branch: extradoc
Changeset: r3999:a88377852aa3
Date: 2012-01-03 14:30 +0100

Log:	merge

diff --git a/blog/draft/matplotlib.rst b/blog/draft/matplotlib.rst
new file mode 100644
--- /dev/null
+++ b/blog/draft/matplotlib.rst
@@ -0,0 +1,86 @@
+Plotting using matplotlib from PyPy
+**Big fat warning** This is just a proof of concept. It barely works. There are
+missing pieces left and right, which were replaced with hacks so I can get this
+to run and prove it's possible. Don't try this at home, especially your home.
+You have been warned.
+There has been a lot of talking about PyPy not integrating well with the
+current scientific Python ecosystem, and ``numpypy`` (a NumPy reimplementation
+on top of pypy) was dubbed "a fancy array library". I'm going to show that
+integration with this ecosystem is possible with our design.
+First, `the demo`_::
+  #!/usr/bin/env pypy
+  # numpy, pypy version
+  import numpypy as numpy
+  # DRAGONS LIVE THERE (fortunately hidden)
+  from embed.emb import import_mod
+  pylab = import_mod('matplotlib.pylab')
+  if __name__ == '__main__':
+      a = numpy.arange(100, dtype=int)
+      b = numpy.sin(a)
+      pylab.plot(a, b)
+      pylab.show()
+And you get:
+   XXX pic
+Now, how to reproduce it:
+* You need a PyPy without cpyext, I did not find a linker that would support
+  overriding symbols. Right now there are no nightlies like this, so you have
+  to compile it yourself, like::
+    ./translate.py -Ojit targetpypystandalone.py --withoutmod-cpyext
+  That would give you a PyPy that's unable to load some libraries like PIL, but
+  perfectly working otherwise.
+* Speaking of which, you need a reasonably recent PyPy.
+* The approach is generally portable, however the implementation has been
+  tested only on 64bit linux. Few tweaks might be required.
+* You need to install python2.6, the python2.6 development headers, and have
+  numpy and matplotlib installed on that python.
+* You need a checkout of my `hacks directory`_ and put embedded on your
+  ``PYTHONPATH``, your pypy checkout also has to be on the ``PYTHONPATH``.
+Er wait, what happened?
+What didn't happen is we did not reimplement matplotlib on top of PyPy. What
+did happen is we embed CPython inside of PyPy using ctypes. We instantiate it.
+and follow the `embedding`_ tutorial for CPython. Since numpy arrays are not
+movable, we're able to pass around an integer that's represents the memory
+address of the array data and reconstruct it in the embedded interpreter. Hence
+with a relatively little effort we managed to reuse the same array data on both
+sides to plot at array. Easy, no?
+This approach can be extended to support anything that's not too tied with
+python objects. SciPy and matplotlib both fall into the same category
+but probably the same strategy can be applied to anything, like GTK or QT.
+It's just a matter of extending a hack into a working library.
+To summarize, while we're busy making numpypy better and faster, it seems
+that all external libraries on the C side can be done using an embedded Python
+interpreter with relatively little effort. To get to that point, I spent
+a day and a half to learn how to embed CPython, with very little prior
+experience in the CPython APIs. Of course you should still keep as much as
+possible in PyPy to make it nice and fast :)
+.. _`hacks directory`: https://bitbucket.org/fijal/hack2
+.. _`the demo`: https://bitbucket.org/fijal/hack2/src/default/embed/embed/matplotwrapper.py
+.. _`embedding`: http://docs.python.org/extending/embedding.html
diff --git a/blog/draft/pycon-2012-teaser.rst b/blog/draft/pycon-2012-teaser.rst
new file mode 100644
--- /dev/null
+++ b/blog/draft/pycon-2012-teaser.rst
@@ -0,0 +1,38 @@
+Come see us at PyCon 2012
+`PyCon 2012`_ is coming up in just a few short months, and PyPy will be well
+represented there.  We'll be delivering a tutorial, two talks, plus we'll be
+around for the sprints.
+Here are the abstracts for the tutorials and talks:
+* **How to get the most out of your PyPy**, by Maciej Fijalkowski, Alex Gaynor
+  and Armin Rigo: For many applications PyPy can provide performance benefits
+  right out of the box. However, little details can push your application to
+  perform much better. In this tutorial we'll give you insights on how to push
+  PyPy to it's limits. We'll focus on understanding the performance
+  characteristics of PyPy, and learning the analysis tools in order to maximize
+  your applications performance. *This is the tutorial.*
+* **Why PyPy by example**, by Maciej Fijalkowski, Alex Gaynor and Armin Rigo:
+  One of the goals of PyPy is to make existing Python code faster, however an
+  even broader goal was to make it possible to write things in Python that
+  previous would needed to be written in C or other low-level language. This
+  talk will show examples of this, and describe how they represent the
+  tremendous progress PyPy has made, and what it means for people looking to
+  use PyPy.
+* **How the PyPy JIT works**, by Benjamin Peterson: The Python community is
+  abuzz about the major speed gains PyPy can offer pure Python code. But how
+  does PyPy JIT actually work? This talk will discuss how the PyPy JIT is
+  implemented. It will include descriptions of the tracing, optimization, and
+  assembly generation phases. I will demonstrate each step with a example loop.
+If you have any questions let us know!  We look forward to seeing people at
+PyCon and chatting about PyPy and the entire Python ecosystem.
+See you there,
+Maciej Fijalkowski, Alex Gaynor, Benjamin Peterson, Armin Rigo, and the entire PyPy team
+.. _`PyCon 2012`: https://us.pycon.org/2012/
diff --git a/blog/draft/screen0.png b/blog/draft/screen0.png
new file mode 100644
index 0000000000000000000000000000000000000000..0f6d5f2e8bdb961c6fc893e6d745910f98792684
GIT binary patch


diff --git a/planning/micronumpy.txt b/planning/micronumpy.txt
--- a/planning/micronumpy.txt
+++ b/planning/micronumpy.txt
@@ -1,10 +1,6 @@
-- add in numpy.generic and the various subclasses, use them in returning
-  instances from subscripting (and possibly internally), also make them valid
-  for the dtype arguments (numpy-dtype-refactor branch)
 - astype
 - a good sort function
@@ -13,16 +9,20 @@
 - endianness
-- scalar types like numpy.int8 (numpy-dtype-refacotr branch)
-- add multi-dim arrays (numpy-multidim-shards branch)
-  - will need to refactor some functions
 - frompyfunc to create ufuncs from python functions
 - more ufuncs
-- arange/linspace/other ranges
+- linspace/other ranges
-- numpy.flatiter array.flat and friends
+- more attributes/methods on numpy.flatiter
+- axis= parameter to various methods
+- expose ndarray.ctypes
+- subclassing ndarray (instantiating subcalsses curently returns the wrong type)
+  * keep subclass type when slicing, __array_finalize__
+  * ndarray.view
diff --git a/sprintinfo/leysin-winter-2011/announcement.txt b/sprintinfo/leysin-winter-2012/announcement.txt
copy from sprintinfo/leysin-winter-2011/announcement.txt
copy to sprintinfo/leysin-winter-2012/announcement.txt
--- a/sprintinfo/leysin-winter-2011/announcement.txt
+++ b/sprintinfo/leysin-winter-2012/announcement.txt
@@ -1,30 +1,23 @@
-         PyPy Leysin Winter Sprint (16-22nd January 2011)
+         PyPy Leysin Winter Sprint (15-22nd January 2012)
 The next PyPy sprint will be in Leysin, Switzerland, for the
-seventh time.  This is a fully public sprint: newcomers and topics
+eighth time.  This is a fully public sprint: newcomers and topics
 other than those proposed below are welcome.
 Goals and topics of the sprint
-* Now that we have released 1.4, and plan to release 1.4.1 soon
-  (possibly before the sprint), the sprint itself is going to be
-  mainly working on fixing issues reported by various users. Of
-  course this does not prevent people from showing up with a more
-  precise interest in mind.  If there are newcomers, we will gladly
-  give introduction talks.
+* Py3k: work towards supporting Python 3 in PyPy 
-* We will also work on polishing and merging the long-standing
-  branches that are around, which could eventually lead to the
-  next PyPy release.  These branches are notably:
+* NumPyPy: work towards supporting the numpy module in PyPy 
-  - fast-forward (Python 2.7 support, by Benjamin, Amaury, and others)
-  - jit-unroll-loops (improve JITting of smaller loops, by Hakan)
-  - arm-backend (a JIT backend for ARM, by David)
-  - jitypes2 (fast ctypes calls with the JIT, by Antonio).
+* JIT backends: integrate tests for ARM; look at the PowerPC 64;
+  maybe try again to write an LLVM- or GCC-based one
+* STM and STM-related topics; or the Concurrent Mark-n-Sweep GC
 * And as usual, the main side goal is to have fun in winter sports :-)
   We can take a day off for ski.
@@ -33,8 +26,9 @@
 Exact times
-The work days should be 16-22 January 2011.  People may arrive on
-the 15th already and/or leave on the 23rd.
+The work days should be 15-21 January 2011 (Sunday-Saturday).  The
+official plans are for people to arrive on the 14th or the 15th, and to
+leave on the 22nd.
 Location & Accomodation
@@ -56,13 +50,14 @@
 expensive) and maybe the possibility to get a single room if you really want
-Please register by svn:
+Please register by Mercurial::
-  http://codespeak.net/svn/pypy/extradoc/sprintinfo/leysin-winter-2011/people.txt
+  https://bitbucket.org/pypy/extradoc/
+  https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2012
-or on the pypy-sprint mailing list if you do not yet have check-in rights:
+or on the pypy-dev mailing list if you do not yet have check-in rights:
-  http://codespeak.net/mailman/listinfo/pypy-sprint
+  http://mail.python.org/mailman/listinfo/pypy-dev
 You need a Swiss-to-(insert country here) power adapter.  There will be
 some Swiss-to-EU adapters around -- bring a EU-format power strip if you
diff --git a/sprintinfo/leysin-winter-2012/people.txt b/sprintinfo/leysin-winter-2012/people.txt
new file mode 100644
--- /dev/null
+++ b/sprintinfo/leysin-winter-2012/people.txt
@@ -0,0 +1,58 @@
+People coming to the Leysin sprint Winter 2011
+People who have a ``?`` in their arrive/depart or accomodation
+column are known to be coming but there are no details 
+available yet from them.
+==================== ============== =======================
+    Name              Arrive/Depart     Accomodation 
+==================== ============== =======================
+Armin Rigo                          private
+David Schneider      17/22          ermina
+==================== ============== =======================
+People on the following list were present at previous sprints: 
+==================== ============== =====================
+       Name          Arrive/Depart  Accomodation 
+==================== ============== =====================
+Antonio Cuni         ?              ?
+Michael Foord        ?              ?
+Maciej Fijalkowski   ?              ?
+David Schneider      ?              ?
+Jacob Hallen         ?              ?
+Laura Creighton      ?              ?
+Hakan Ardo           ?              ?
+Carl Friedrich Bolz  ?              ?
+Samuele Pedroni      ?              ?
+Anders Hammarquist   ?              ?
+Christian Tismer     ?              ?
+Niko Matsakis        ?              ?
+Toby Watson          ?              ?
+Paul deGrandis       ?              ?
+Michael Hudson       ?              ?
+Anders Lehmann       ?              ?
+Niklaus Haldimann    ?              ?
+Lene Wagner          ?              ?
+Amaury Forgeot d'Arc ?              ?
+Valentino Volonghi   ?              ?
+Boris Feigin	     ?              ?
+Andrew Thompson      ?              ?
+Bert Freudenberg     ?              ?
+Beatrice Duering     ?              ?
+Richard Emslie       ?              ?
+Johan Hahn           ?              ?
+Stephan Diehl        ?              ?
+Alexander Schremmer  ?              ?
+Anders Chrigstroem   ?              ?
+Eric van Riet Paap   ?              ?
+Holger Krekel        ?              ?
+Guido Wesdorp        ?              ?
+Leonardo Santagada   ?              ?
+Alexandre Fayolle    ?              ?
+Sylvain Th&#65533;nault     ?              ?
+==================== ============== =====================
diff --git a/talk/iwtc11/benchmarks/convolution/convolution.py b/talk/iwtc11/benchmarks/convolution/convolution.py
--- a/talk/iwtc11/benchmarks/convolution/convolution.py
+++ b/talk/iwtc11/benchmarks/convolution/convolution.py
@@ -57,6 +57,19 @@
                 self[x, y] = data[y][x]
         return self
+class NumpyArray(Array2D):
+    def __init__(self, w, h):
+        self.width = w
+        self.height = h
+        import numpypy
+        self.data = numpypy.zeros([h, w], 'd')
+    def __getitem__(self, (x, y)):
+        return self.data[y, x]
+    def __setitem__(self, (x, y), val):
+        self.data[y, x] = val
 def _conv3x3(a, b, k):
     assert k.width == k.height == 3
     for y in xrange(1, a.height-1):
@@ -88,6 +101,13 @@
         _conv3x3(a, b, Array2D(3,3))
     return 'conv3x3(Array2D(%sx%s))' % tuple(args)
+def conv3x3_numpy(args):
+    a = NumpyArray(int(args[0]), int(args[1]))
+    b = NumpyArray(a.width, a.height)
+    for i in range(10):
+        _conv3x3(a, b, NumpyArray(3,3))
+    return 'conv3x3(NumpyArray(%sx%s))' % tuple(args)
 def dilate3x3(args):
     a = Array2D(int(args[0]), int(args[1]))
     b = Array2D(a.width, a.height)
diff --git a/talk/iwtc11/benchmarks/image/io.py b/talk/iwtc11/benchmarks/image/io.py
--- a/talk/iwtc11/benchmarks/image/io.py
+++ b/talk/iwtc11/benchmarks/image/io.py
@@ -1,4 +1,6 @@
 import os, re, array
+from subprocess import Popen, PIPE, STDOUT
 def mplayer(Image, fn='tv://', options=''):
     f = os.popen('mplayer -really-quiet -noframedrop ' + options + ' ' 
@@ -19,18 +21,18 @@
     def view(self, img):
         assert img.typecode == 'B'
         if not self.width:
-            self.mplayer = os.popen('mplayer -really-quiet -noframedrop - ' +
-                                    '2> /dev/null ', 'w')
-            self.mplayer.write('YUV4MPEG2 W%d H%d F100:1 Ip A1:1\n' %
-                               (img.width, img.height))
+            w, h = img.width, img.height
+            self.mplayer = Popen(['mplayer', '-', '-benchmark',
+                                  '-demuxer', 'rawvideo',
+                                 '-rawvideo', 'w=%d:h=%d:format=y8' % (w, h),
+                                 '-really-quiet'],
+                                 stdin=PIPE, stdout=PIPE, stderr=PIPE)
             self.width = img.width
             self.height = img.height
-            self.color_data = array.array('B', [127]) * (img.width * img.height / 2)
         assert self.width == img.width
         assert self.height == img.height
-        self.mplayer.write('FRAME\n')
-        img.tofile(self.mplayer)
-        self.color_data.tofile(self.mplayer)
+        img.tofile(self.mplayer.stdin)
 default_viewer = MplayerViewer()
diff --git a/talk/iwtc11/licm.pdf b/talk/iwtc11/licm.pdf
new file mode 100644
index 0000000000000000000000000000000000000000..ff2a7bf547f542771702ac86ea8531f8ba16cc28
GIT binary patch


diff --git a/talk/sea2012/abstract.rst b/talk/sea2012/abstract.rst
new file mode 100644
--- /dev/null
+++ b/talk/sea2012/abstract.rst
@@ -0,0 +1,18 @@
+Fast numeric in Python - NumPy and PyPy
+Python increasingly is being utilized as a powerful scientific
+processing language. It successfully has been used as a glue language
+to drive simulations written in C, Fortran or the array
+manipulation language provided by the NumPy package.  Originally
+Python only was used as a glue language because the original Python
+implementation was relatively slow. With the recent progress in the
+PyPy project that is showing significant performance
+improvements in each release, Python is nearing performance comparable
+to native C language implementations. In this talk I will
+describe three stages: how to use it right now, in the near future and
+(xxx give before this line a hint that "we" or "our" means "pypy developers")
+our plans to provide a very robust infrastructure for implementing
+numerical computations. I also will spend some time exploring ideas
+how dynamic compilation eventually can outperform static compilation
+and how a high-level language helps accomplish this.

More information about the pypy-commit mailing list