[Numpy-discussion] Fwd: YASS (Yet Another Success Story)

Emanuel Woiski woiski at gmail.com
Sat Jun 27 17:08:58 EDT 2009

Hstória de sucesso

---------- Forwarded message ----------
From: k3xji <sumerc at gmail.com>
Date: 2009/6/20
Subject: YASS (Yet Another Success Story)
To: python-list at python.org

Hi all,

Started a project year ago with hard goals in mind : Developing a game
server which can handle thousands of clients simultaneously.System
must be stable, scalable, efficient, and if the client need to migrate
the server to another OS, this will be a matter of time, if possible
without changing any code. Also, the system should be controlled via
database operations, meaning some specific DB changes should be
determined by the system on-the-fly and necassary processing should be
done at runtime(Client timeout values, adding new game lobbies..etc..)
And all of this should be a one-man project dues to the fact that the
project is a volunteer work, thus is separate from our ordinary jobs,
there is no budget:). This was 8 months ago, and I still remember we
are talking with my colleguae to decide which programming language is
better for our requirements. We wrote the tradeoffs of different
languages on board and compare our objectives against them by giving
numbers. I remember, the last three standing programming languages
are: C++, Java, and Python. We decided that code maintability and
stability is cheaper than native code efficiency and also easy porting
is important. C++ is only successfull on efficiency but maintaining
the code, even if we use 3rd party libraries is a nightmare,
especially in a massive multithreaded project like a game server.
Also, we decided a game server's bottleneck is on network I/O other
than the actual processing which means there should be not so much
performance degradation between C++ and an interpreted language. So
the final is between Java and Python. I know Java has been adding very
good optimizations for interpreter efficiency over the years(JIT
compiling..etc.) And I remember somewhere reading that Java is 10
times faster than Python. But Python, on the other hand is a perfect
language and very simple which means the project may be completed
sooner and efficiency again will be not a so much problem as the
bottleneck should be on network I/O if the server is coded correctly.
Although we have made this decision, we still had some doubts but we
move on with Python.
I don't know Python at the time and only coded few simple projects
with it.
 I started with reading python socket/threading tutorials, then I
downloaded and read actual Python code to get deeper and deeper. This
was 10 months ago. Now we have been able to implement the game server
with Python using MySQL as a backend DB and currently 5000 players
(increasing) are playing multiplayer games with it with a system load
of 0.4-0.8(see Unix getloadavg output) on a 1.8 Ghz Dual Xeon
processor. Let me give some benchmarks, please note that one thing I
learn through this journey is that testing/benchmarking a server
software is really difficult and maybe in some conditions impossible
thing to do correctly as inactive client connections may cause
indeterministic behavior and also WAN latencies are very different
than LAN latencies..etc. Anyway, we had to do a benchmark before going
with this server on live. We installed a UnrealIRCd server which is
also using select() as an I/O paradigm just as we are and supporting
many of the features we have. This server is a IRC server written in C
and being written since nearly 10 years. We connect thousands of
clients and do simple messaging between them. I don't have an official
test/benchmark result or a nice graph for this, however the system
running UnrealIRCD is locked up when processing 4 Mb/sec of data,
where our 6 month old Python server is locking up the system resources
after 3.5 Mb/sec. We have done other tests including stressing server
accept(), adding inactive connections but I am really not %100
confident about the  test environments and  do not want to give you
false impression. But we decided that this 0.5 Mb factor is still not
enough for us and we move on to inspect what is really causing the
bottlenecks and how to optimize them. Here, to be honest, I cannot see
very uch help from Python. Profiling/Optimization is the only place I
am upset with Python. Here is the requirements for the profiler: I
want to be able to enable/disable profiler tool on the fly without
requiring any kind of restart and the profiler should be multithreaded
(thus thread-safe). I decided that cProfiler and other profiler tools
are in need of some updates to fulfill my reqırements, which makes me
write my own profiler. It took me 3 days to come up with a profiler
that satisfies my requirements, this profiler will profile only my
code, meaning the server code to avoid performance degradation and
timings will need not be nested-aware and timing precision is not very
important (other than secs). I have implemented the profiler with the
help of decorators and started profiling my code on-live. Here comes
the another part of Python that is some kind of shady: optimization.
After profiling the code, it turns out most of the time is spent on
the following:
1) Selecting/Deselecting interest sets for select()
2) Unnecessary function calls. (for integrating some OOP concepts, I
think we have overlooked the performance factor and there really are
some functions which can be inlined but declared as a function)
3) Redundant try-except's in all over place(Again our fault to make
the system stable, we have put some debug purposed-assert like try-
excepts in the main server flow.)

After, looking on these subjects for 2 weeks, we came up with
solutions for all of them, and we see that it is working approximately
5 times faster. This test is done in live, we run two server processes
which are all running on a predefined CPU(means CPU affinity of the
process is set before hand to avoid mis-interpreting system diagnose
tools like top(1)). When same client number is connected to both
servers, the second one is giving 5 times better values in top and ps
tools. Also, the processing time (meaning the server loop except the
select() call) is also 10 times faster than before. Just one note
about optimizing Python code: do not optimize Python code based on
your assumptions, just go and test if it really runs faster. I don't
want to go to details of this hint, but believe me making Python code
optimized may be very very tricky.

It is then I decided to write up here this as a success story, as I am
very newcomer to Python but come up with a nearly commercial product
in a very short period of time and I don't think this is about my
personal characteristics and intelligence or so:), as I am old enough
to know/meet that there are much much more brilliant people than I am
and they also have similar experiences with Python.

So, one last note: every software project goes same tasks as above
often much much more officially and carefully, I would suggest
managers to see that just do not listen to the ordinary brain-washes.
Python is a great choice for easy developing, easy debugging, easy
maintaining and most importantly very very time-friendly. Of course
there will be tasks .n which Python is suitable, but hey, if it Python
is in the list, take it seriously.


Special thanks to people in this group who had answered my silly
Python questions along the way.

Sümer Cip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20090627/6187013e/attachment.html>

More information about the NumPy-Discussion mailing list