[Python-Dev] Runtime forks through monkeypatching (was Re: Use function names instead of functions for os.supports_dir_fd?)

[Nick Coghlan]

On Fri, Jul 20, 2012 at 4:48 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 18 Jul 2012 02:26:14 +0200
> Victor Stinner <victor.stinner at gmail.com> wrote:
>
>> >> Monkey patching is a common practice in Python. test_os.py replaces
>> >> os.exec*() functions temporary for example.
>> >
>> > Perhaps for testing, but I don't think monkey-patching is common in
>> > production code. Perhaps you are thinking of Ruby :)
>>
>> The gevent library does monkey-patch os.fork (and time.sleep and many
>> other functions), but gevent is maybe not ready for production? :-)
>
> Extensive monkey-patching of core OS functions would certainly make me
> weary of using such a third-party library, even if it claims to be
> "serious".

gevent is one of the well-behaved players in that game - it doesn't
monkeypatch anything unless you explicitly tell it to [1]. You do need
to have some idea what you're doing if using the monkey patching
aspect (as not all third party modules will behave well in that case -
just as not all third party modules behave well on other Python
implementations), but you can also just use gevent as a normal library
without monkey patching anything (however, adhering strictly to that
rule leaves you in the same situation as Twisted, where modules that
use the standard blocking APIs are completely useless to you. Thus, if
your goal is "pure async" code without any need for monkeypatching,
Twisted is a much better bet than gevent, just because it's been
playing that game for longer and has more fully async aware components
available).

That's why I call features like the gevent.monkey module "runtime
forks" - when they're implemented well, they effectively give you two
Python implementations without the complexity of actually having
parallel Python installations (compare the complexity of using
Stackless, a traditional fork, vs gevent.monkey, a runtime fork -
there's a reason the core greenlet component was lifted out of
Stackless and made available as a CPython extension module).
Applications can decide at startup which Python variant (standard
CPython, or CPython+gevent) they want to use without affecting other
Python applications installed on the system and without needing to
duplicating all installed Python modules into a second interpreter.

It's only when libraries monkeypatch standard interfaces *implicitly*
as a side effect of import that you start to get seriously broken
"action at a distance" behaviour. Or, as happened with distutils, you
get a third party project (setuptools) effectively freezing
implementation details of the stdlib version - when monkeypatching
starts delving too deep into implementation details, it's time to
switch to a traditional fork and start figuring out better ways to
solve the problem (which process is now grinding away on the
distutils2/packaging front).

Monkeypatching is a powerful tool - used well, it lets you achieve
impressive things that would otherwise be substantially more difficult
to either create in the first place or maintain in the long run. As
with many powerful tools though, used badly it can lead to a lot of
pain. Static languages like Java decide the risk of action at a
distance from monkeypatching isn't worth the power it gives you to
work around issues in third party libraries without resorting to
creating a traditional fork, so they strictly enforce access controls
and have completely closed class definitions. Python permits
monkeypatching at a language level (because when you need it, you
really need it), but discourages it a social level (because there is
usually a less magical technique, such as creating and using a wrapper
or subclass, that will achieve the desired effect).

Cheers,
Nick.

[1] http://www.gevent.org/intro.html#monkey-patching

P.S. for a non-monkeypatching approach to reusing a synchronous
library when creating an asynchronous equivalent, take a look at
10gen's attempt to create a *maintainable* Tornado-compatible async
wrapper for PyMongo by using greenlets ([2][3]). It's a clever idea,
but still a lot more work than just using PyMongo itself with gevent
if you're only interested in the "async" part rather than the
"Tornado-compatible" part.

[2] http://emptysquare.net/blog/motor-internals-how-i-asynchronized-a-synchronous-library/
[3] http://emptysquare.net/blog/motor-four-strategies-for-maintainability/


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia