[pypy-dev] Parallella open hardware platform

[John Camara]

Hi Armin,

It's even worse I'm asking you to support <X> and I don't even need it.

When I posted this thread it was getting rather long and unfortunately I
didn't really make all the points I wanted to make.  At this point, and
even for some time now PyPy has a great foundation but it's use remains
low.  Every now and then it's good to step back a little bit and reflect on
the current situation and come up with a strategy that helps the project's
popularity grow.  I know that PyPy has done things to help with the growth
such as writing blog posts, being quick to fix bugs, helping others with
their performance issues and even rapidly adding optimizations to PyPy,
presenting at conferences, and often actively being engaged in commenting
any posts or comments made about PyPy.

So PyPy is doing a lot of things right to help it's PR but yet there is
this issue of slow growth.  Now we know what the main issue is with it's
growth is the fact that the Python ecosystem relies on a lot of libraries
that use the CPython API and PyPy just doesn't have full support for this
interface.  I understand the reasons why PyPy is not going to support the
full interface and PyPy has come up with the cffi library as a way to
bridge the gap.  And of course I don't expect the PyPy project to take on
the responsibility of porting all the popular 3rd party libraries that use
the CPython API to cffi.  It's going to have to be a community effort.  One
thing that could help would be more marketing of cffi as very few Python
developers know it exists.  But that along is not going to be enough.

History tells us that most successful products/projects that become popular
do so by first supporting the needs of some niche market.  As time goes by
that niche market starts providing PR that helps other markets to discover
the product/project and the cycle can sometimes continue until there is
mass adoption.  Now when PyPy started to place a focus on NumPy I had hoped
that the market it serves would turn out to be the market that would help
PyPy grow.  But at this point in time it does not appear like that is going
to happen.

For a while I have been trying to think of a niche market that maybe
helpful. But to do so you have to consider the current state of PyPy which
means eliminating markets that heavily rely on libraries that use the
CPython API, also going to avoid the NumPy market as that's currently being
worked on, there is the mobile market but that's a tough one to get into,
maybe the gaming market could be a good one, etc.  It turns out with the
current state of PyPy many markets need to be eliminated if you looking for
one that is going to help with growth. The parrallella project on the other
hand looks like it could be a promising one and I'll share so thoughts a
little later in this post as to why I feel this way.

Right now you have been putting a lot of effort into STM in which your
trying to solve what is likely the biggest challenge that the developer
community is facing.  That is how to write software that effective
leverages many cores in a way that is straight forward and in the spirit of
Python. When you solve this problem and I have the faith that you will,
most would think that it would cause PyPy's popularity to sky rocket.

What most likely will happen is that PyPy gets a temporary boost in
popularity as there is another lesson in history to be concerned about.
 Often the first to solve a problem does not become popular in the long
run.  As usually the first to solve the problem does so via a unique
solution but once people start using it issues with the approach gets
discovered.  Then often many others will use the original solution solution
as a starting point and modify it to eliminate these new issues.  Then one
of the second generation solutions ends up being the defacto standard.

Now PyPy is able to move fairly quickly in terms of implementing new
approaches so it may in fact be able to compete just fine against other 2nd
generation solutions. But there may be some benefits to exposing STM for a
smaller market to help PyPy buy some additional time before releasing it as
a solution for the general developer community.

So why the Parallella project.  Well I think it can be helpful in a number
of ways. First I don't believe that this market is going to need much from
the libraries that use the CPython APIs.  Many who are in this market are
used to having to program for embedded systems and are more likely have the
skills to help out the PyPy project in a number of areas and would likely
also have a financial incentive to contribute back to PyPy such as helping
keep various back ends up to date such as Arm, PPC, and additional
architectures.  Some in this market are used to using a number of graphical
languages to program their devices but unfortunately for them some of the
new products that need to enter the market can't be built fully with these
graphical languages.  Well with the PyPy framework it's possible for them
to implement a VM for that graphical language and be able to create
products that contain elements programmed in both the graphical languages
as well as text based languages.  Also the VMs on many embedded systems are
typically simple and don't have a JIT.  PyPy can help with this but I don't
believe any one who maintains these VMs are aware of the PyPy project.

As far as STM is concerned, working with embedded systems will force
finding solutions to the many issue that arise with various hardware
architectures which would help STM become a more general solution.  Right
now your currently writing STM in a way that will support multiple cores on
a single processor well.  I know you have to start some where.  But soon
you will have to deal with issues that arise once you span to multiple
processors such as dealing more often with the slower L3 cache and it sync
issues and local vs remote memory issues.  But on the embedded side you
have to deal with processors of multiple architectures on the same system
plus FPGAs as well as having to consider the various issues that a arise
from the various buses involved which makes the STM problem quit a bit
harder in how it gets optimized to handle all these variations.  Of course
many of these same issues exists if you want to have STM support GPUs in a
normal computing device.  The embedded side just
adds additional complications as they come in more complex configurations.

The Raspberry Pi has become popular, as many want to hack on these devices
and the Raspberry Pi happens to be the first devices that is both cheap and
allows programming at a high level.  Previously if you wanted cheap it
meant you needed to program using low level approach or you had to buy an
expensive solution to program at a high level.  Many who get interested in
the Raspberry Pi soon find them self in the position where they have an
idea and want to create a product to sell.  But they realize you can't use
a Raspberry Pi for production as it missing many features they would be
required but they also like the idea of programming at a high level but the
traditional embedded systems that support this may be too expensive for
their product.  That's where the Parallella project comes into play.  They
see there is a market for a low cost devices that can be programmed with
higher level tools to build production systems.

This market values programming at a high level and would highly appreciate
being able to program them in Python.  They also have a need to support
multi cores and thus could use STM and it would be incredibly usefully if
the STM approach could seamlessly support multiple architectures.  There is
a lot of value here for the companies that want to produce these devices
and PyPy should try to tap into it.

This new market segment using these low cost devices are going to have a
large impact and also will play a role in the manufacturing revolution that
is about to take place.  This manufacturing revolution is likely to be on
the same scale as the Internet revolution.  Just think about what the
effect 3D printing is going to have.  It will be huge.  PyPy getting a foot
hold into this market before it takes off would be huge for PyPy as well as
in general for Python.

Also there are some big players who currently sell these more expensive
embedded systems who are not going to be happy about these cheaper
alternatives and are also going to want a piece of the action.  I think for
many of them who may not be able to quickly change their development
and run-time processes may decide it's much easier for them to port their
VMs over to PyPy to get into the action.

Hopefully this gives some better insight as to why I feel it may be a
good strategy to consider supporting the Parallella project.
 The possibility of getting a foot hold into a market that is about to take
off doesn't come around too often.  All I know is if PyPy would like
to support this market right now is the best time to get started.

This might be the ticket PyPy needs to gets it growth up which could then
lead to additional markets taking notice and more of the Python ecosystem
becoming compatible with PyPy.

Of course this is just my opinion and maybe someone else could come up with
another strategy that can help PyPy grow faster. Even an Open Source
project can use a strategy.

John


On Tue, Feb 5, 2013 at 9:47 AM, Armin Rigo <arigo at tunes.org> wrote:

> Hi John,
>
> Sorry if I misread you, but you seem to be only saying "it would be
> nice if the PyPy team worked on the support for <X> rather than <Y>".
> While this might be true under some point of view, it is not
> constructive.  What would be nice is if *you* seriously proposed to
> work on <X>, or helped us raise commercial interest, or otherwise
> contributed towards <X>.  If you're not up to it, and nobody steps up,
> then it's the end of the story (but thanks anyway for the nice
> description of Parallela).
>
>
> A bientôt,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20130205/17426218/attachment-0001.html>