[BangPypers] When *not* to use Python

Dhananjay Nene dhananjay.nene at gmail.com
Tue Oct 26 09:01:18 CEST 2010


On Tue, Oct 26, 2010 at 11:47 AM, Anand Balachandran Pillai <
abpillai at gmail.com> wrote:

> On Mon, Oct 25, 2010 at 7:33 AM, Dhananjay Nene <dhananjay.nene at gmail.com
> >wrote:
>
> > On Wed, Oct 20, 2010 at 4:57 PM, Noufal Ibrahim <noufal at gmail.com>
> wrote:
> >
> > >
> > >
> > > Assuming that most of the people here are mostly python enthusiasts or
> > > learners, I'm wondering when you would *not* use Python. Let's not
> > > conflate this with Open Source/Closed Source etc.
> > >
> > >
> > Python is the language which has (what in business terms is coined as)
> the
> > right of first refusal.
> >
> > ie. evaluate something whether it really makes sense in python, and if
> not
> > in python then start exploring what else.
> >
> > The primary reasons which may encourage me to use a language other than
> > python are :
> >
> > a. Need for raw CPU power. Yes there are a few of such use cases. When
> > purely CPU driven and when you really really need the CPU badly (which is
> > probably less than 5% cases) python performance is really poor.
> > b. Need for high amounts of multithreading even while being largely CPU
> > bound.
> >
> > Yes - both situations can be handled by escaping to C from python, but I
> > would much rather handle these in Java (or languages based on JVM) rather
> > than C unless I am in a very constrained hardware situation.
> >
> > On a separate note I do believe as we start heading towards the 32 core
> > situation (say in 5 yrs time) - there better exist a better solution than
> > the GIL :)
> >
> >
>  I think we should move from the thinking that concurrency <= threading
>  and move over to better solutions in languages like Python which have
>  better ways of solving the concurrency problem than actually having to
>  create OS level threads.
>

Threading is not the only solution to address concurrency but it is a valid
one. The only issue I have with the statement above is that it seems to
suggest that the move will address all use cases. It simply will not. At the
same time I do agree that imagining threading to be the one and only
solution to address concurrency is equally inadvisable.


>
>  If you look at it differently, concurrency is the real problem and
> threading
>  is just one approach to it which is kind of resource intensive and doesn't
>  work great in Python due to the GIL. But instead of GIL bashing, one
> should
>  look at other approaches to concurrency such as generator based
> co-operative
>  "threading" and also look at numerous other concurrency libraries in
> Python
>  before deciding that going the way of actual threads is his solution.
>

Here's an example of what I had done once for an extremely extremely high
performance application (million+ financial transactions per hour). There
were layers of caches (with some cache coordination to ensure consistency).
It was the high speed caches and cache access and which really helped
achieve that performance. And that was possible because I had a very large
number of threads sharing some of the caches (some caches were thread
level). I cannot imagine making such an application work in a multi process
environment. While I did not evaluate cooperative threading, I suspect that
would also introduce its own overheads.

When one really really needs threads, GIL sucks. What I am saying is that
GIL bashing is not unreasonable given the context of specific use cases (we
were discussing situations where one would prefer to not use python).


>
>  Python should go the direction shown by languages such as Erlang which has
>  true user-space multitasking with light-weight "processes". It already has
>  the infrastructure for it with generators etc but support for true
> multitasking
>  at user level should be part of the language rather than added on by
>  third party libraries. Right now if one decides to explore Python
> concurrency,
>  the scene is truly confusing and a little scary, since there are too many
>  approaches out there which also share a lot of similarities.
>
>
I agree (go down direction of light weight processes).


>  Hopefully the "futures" have something to offer here :)
>
> Once again agreed.


> --Anand
>
>
>
>
>
>
>
> >
> > > --
> > > ~noufal
> > > http://nibrahim.net.in
> > >
> > > _______________________________________________
> > > BangPypers mailing list
> > > BangPypers at python.org
> > > http://mail.python.org/mailman/listinfo/bangpypers
> > >
> >
> >
> >
> > --
> > --------------------------------------------------------
> > blog: http://blog.dhananjaynene.com
> > twitter: http://twitter.com/dnene
> > _______________________________________________
> > BangPypers mailing list
> > BangPypers at python.org
> > http://mail.python.org/mailman/listinfo/bangpypers
> >
>
>
>
> --
> --Anand
> _______________________________________________
> BangPypers mailing list
> BangPypers at python.org
> http://mail.python.org/mailman/listinfo/bangpypers
>



-- 
--------------------------------------------------------
blog: http://blog.dhananjaynene.com
twitter: http://twitter.com/dnene


More information about the BangPypers mailing list