Howdy all,
I am preparing a PEP, and corresponding reference implementation, for
a standard implementation of the steps needed to turn a program into a
well-behaved Unix daemon.
This message is a call for comments, support, and suggestions, prior
to submitting this PEP to the editor.
:PEP: XXX
:Title: Standard daemon process library
:Version: 0.1
:Last-Modified: 2009-01-26 08:44
:Author: Ben Finney <ben+python(a)benfinney.id.au>
:Status: Draft
:Type: Standards Track
:Content-Type: text/x-rst
:Created: 2009-01-26
:Python-Version: 3.1
:Post-History:
========
Abstract
========
Writing a program to become a well-behaved Unix daemon is somewhat
complex and tricky to get right, yet the steps are largely similar for
any daemon regardless of what else the program may need to do.
This PEP introduces a module to the Python standard library that
provides a simple interface to the task of becoming a daemon process.
.. contents::
..
Table of Contents:
Abstract
Specification
Example usage
Interface
``Daemon`` objects
``DaemonError`` objects
Motivation
Rationale
Correct daemon behaviour
Reference Implementation
References
Copyright
=============
Specification
=============
Example usage
=============
Simple example of usage::
import daemon
from spam import do_main_program
this_daemon = daemon.Daemon()
this_daemon.start()
do_main_program()
More complex example usage::
import os
import grp
import signal
import daemon
from spam import (
initial_program_setup,
do_main_program,
program_cleanup,
reload_program_config,
)
initial_program_setup()
important_file = open('spam.data', 'w')
interesting_file = open('eggs.data', 'w')
this_daemon = daemon.Daemon()
this_daemon.files_preserve = [important_file, interesting_file]
this_daemon.working_directory = '/var/lib/foo'
this_daemon.umask = 0o002
mail_gid = grp.getgrnam('mail').gr_gid
this_daemon.gid = mail_gid
this_daemon.terminate_callback = program_cleanup
this_daemon.reload_callback = reload_program_config
this_daemon.reload_signals = [signal.SIGHUP, signal.SIGUSR1]
this_daemon.start()
do_main_program()
Interface
=========
A new module, `daemon`, is added to the standard library.
The module defines a class, `Daemon`, used to represent the settings
for a daemon process.
An exception class, `DaemonError`, is defined for exceptions raised
from the module.
``Daemon`` objects
==================
A `Daemon` instance represents the behaviour settings for the process
when it becomes a daemon. The behaviour is customised by setting
attributes on the instance, before calling the `start` method.
The following attributes are defined.
`files_preserve`
:Default: ``None``
List of files that should *not* be closed when starting the
daemon. If ``None``, all open file descriptors will be closed.
Elements of the list are file descriptors (as returned by a file
object's `fileno()` method) or Python `file` objects. Each
specifies a file that is not to be closed during daemon start.
`chroot_directory`
:Default: ``None``
Full path to a directory to set as the effective root directory of
the process. If ``None``, specifies that the root directory is not
to be changed.
`working_directory`
:Default: ``'/'``
Full path of the working directory to which the process should
change on daemon start.
Since a filesystem cannot be unmounted if a process has its
current working directory on that filesystem, this should either
be left at default or set to a directory that is a sensible “home
directory” for the daemon while it is running.
`lockfile_directory`
:Default: ``'/var/run'``
Absolute directory path to contain the daemon's lockfile. If
``None``, the lockfile behaviour for this daemon is skipped.
`lockfile_name`
:Default: ``None``
Base name of the lockfile for this daemon, without directory or
extension. If ``None``, the name is derived from the process
command line.
`umask`
:Default: ``0``
File access creation mask (“umask”) to set for the process on
daemon start.
Since a process inherits its umask from its parent process,
starting the daemon will reset the umask to this value so that
files are created by the daemon with access modes as it expects.
`ignore_signals`
:Default: ``[signal.SIGTTOU, signal.SIGTTIN, signal.SIGTSTP]``
List of signals that the process should ignore (by setting the
signal action to ``signal.SIG_IGN``) on daemon start.
`terminate_signals`
:Default: ``[signal.SIGTERM]``
List of signals that the process should interpret as a request to
terminate cleanly.
`terminate_callback`
:Default: ``None``
Callable to invoke when the process receives any of the
`terminate_signals` signals, before then terminating the process.
`reload_signals`
:Default: ``[signal.SIGHUP]``
List of signals that the process should interpret as a request to
reload runtime configuration.
`reload_callback`
:Default: ``None``
Callable to invoke when the process receives any of the
`reload_signals` signals.
`uid`
:Default: ``None``
The user ID (“uid”) value to switch the process to on daemon start.
`gid`
:Default: ``None``
The group ID (“gid”) value to switch the process to on daemon start.
`prevent_core`
:Default: ``True``
If true, prevents the generation of core files, in order to avoid
leaking sensitive information from daemons run as `root`.
`stdout`
:Default: ``None``
File-like object, open for writing (in append mode, 'w+'), that
will be used as the new value of `sys.stdout`. If it represents an
actual file, it should be listed in `files_preserve` to prevent it
being closed during daemon start. If ``None``, then `sys.stdout`
is not re-bound.
`stderr`
:Default: ``None``
File-like object, open for writing (in append mode, 'w+'), that
will be used as the new value of `sys.stderr`. If it represents an
actual file, it should be listed in `files_preserve` to prevent it
being closed during daemon start. If ``None``, then `sys.stderr`
is not re-bound.
The following methods are defined.
`start()`
:Return: ``None``
Start the daemon. This performs the following steps:
* If the `chroot_directory` attribute is not ``None``:
* Set the effective root directory of the process to that
directory (via `os.chroot`). This allows running the daemon
process inside a “chroot gaol” as a means of limiting the
system's exposure to rogue behaviour by the process.
* If the `lockfile_directory` attribute is not ``None``:
* Look in that directory for a file named '`lockfile_name`.pid';
if it exists, raise a `DaemonError` to prevent multiple
instances of the daemon process.
* Close all open file descriptors, excluding those listed in the
`files_preserve` attribute.
* Change current working directory to the path specified by the
`working_directory` attribute.
* Reset the file access creation mask to the value specified by
the `umask` attribute.
* Detach the current process into its own process group, and
disassociate from any controlling terminal.
This step is skipped if it is determined to be redundant: if the
process was started by `init`, by `initd`, or by `inetd`.
* Set signal handlers as specified by the `ignore_signals`,
`terminate_signals`, `terminate_callback`, `reload_signals`,
`reload_callback` attributes.
* If the `prevent_core` attribute is true:
* Set the resource limits for the process to prevent any core
dump from the process.
* Set the process uid and gid to the true uid and gid of the
process, to relinquish any elevated privilege.
* If the `lockfile_directory` attribute is not ``None``:
* Create the lockfile for this daemon in that directory, by
writing a text line containing the current process ID (“pid”)
to a file named '`lockfile_name`.pid'.
* If either of the attributes `uid` or `gid` are not ``None``:
* Set the process uid and/or gid to the specified values.
* If either of the attributes `stdout` or `stderr` are not
``None``:
* Bind the names `sys.stdout` and/or `sys.stderr` to the
corresponding objects.
`reload()`
:Return: ``None``
Reload the daemon configuration. The meaning of this is entirely
defined by the customisation of this daemon: if the
`reload_callback` attribute is not ``None``, call that object. The
return value is discarded.
`stop()`
:Return: ``None``
Stop the daemon. This performs the following steps:
* If the `terminate_callback` attribute is not ``None``:
* Call that object. The return value is discarded.
* If the `lockfile_directory` attribute is not ``None``:
* Delete the lockfile for this daemon.
* Raise a `SystemExit` exception.
``DaemonError`` objects
=======================
The `DaemonError` class inherits from `Exception`. The module
implementation will raise an instance of `DaemonError` when an error
occurs in processing daemon behaviour.
==========
Motivation
==========
The majority of programs written to be Unix daemons either implement
behaviour very similar to that in the `Specification`_, or are
poorly-behaved daemons by the `Rationale`_.
Since these steps should be much the same in most implementations but
are very particular and easy to omit or implement incorrectly, they
are a prime target for a standard well-tested implementation in the
standard library.
=========
Rationale
=========
Correct daemon behaviour
========================
According to Stevens in [stevens]_ §2.6, a program should perform the
following steps to become a Unix daemon process.
* Close all open file descriptors.
* Change current working directory.
* Reset the file access creation mask.
* Run in the background.
* Disassociate from process group.
* Ignore terminal I/O signals.
* Disassociate from control terminal.
* Don't reacquire a control terminal.
* Correctly handle the following circumstances:
* Started by System V `init` process.
* Daemon termination by ``SIGTERM`` signal.
* Children generate ``SIGCLD`` signal.
The `daemon` package [daemon]_ lists (in its summary of features)
behaviour that should be performed when turning a program into a
well-behaved Unix daemon process. The following are appropriate for a
daemon started once the program is already running:
* Sets up the correct process context for a daemon.
* Behaves sensibly when started by `initd(8)` or `inetd(8)`.
* Revokes any suid or sgid privileges to reduce security risks in case
daemon is incorrectly installed with special privileges.
* Prevents the generation of core files to prevent leaking sensitive
information from daemons run as root (optional).
* Names the daemon by creating and locking a PID file to guarantee
that only one daemon with the given name can execute at any given
time (optional).
* Sets the user and group under which to run the daemon (optional,
root only).
* Creates a chroot gaol (optional, root only).
* Captures the daemon's stdout and stderr and directs them to syslog
(optional).
========================
Reference Implementation
========================
The `python-daemon` package [python-daemon]_.
As of 2009-01-26, the package is under active development and is not
yet a full implementation of this PEP.
==========
References
==========
.. [stevens]
`Unix Network Programming`, W. Richard Stevens, 1994 Prentice
Hall.
.. [daemon]
The (non-Python) `daemon` package,
`<http://www.libslack.org/daemon/>`_ by “raf” <raf(a)raf.org>.
.. [python-daemon]
The `python-daemon` package at the Python Package Index,
`<http://pypi.python.org/pypi/python-daemon>`_. This is the
successor to [bda.daemon]_.
.. [bda.daemon]
The `bda.daemon` package at the Python Package Index,
`<http://pypi.python.org/pypi/bda.daemon>`_. This is an
implementation of [cookbook-66012]_.
.. [cookbook-66012]
Many good ideas were contributed by the community to Python
cookbook recipe 66012, “Fork a daemon process on Unix”
`<http://code.activestate.com/recipes/66012/>`_.
=========
Copyright
=========
This work is hereby placed in the public domain. To the extent that
placing a work in the public domain is not legally possible, the
copyright holder hereby grants to all recipients of this work all
rights and freedoms that would otherwise be restricted by copyright.
--
\ “Pinky, are you pondering what I'm pondering?” “Well, I think |
`\ so (hiccup), but Kevin Costner with an English accent?” —_Pinky |
_o__) and The Brain_ |
Ben Finney
Howdy all,
Many people evidently want a proposal for a “service” interface, and
saw part of what they wanted in the separate discussion of the
“Standard daemon process library”. As discussed there, a Unix daemon
is a quite distinct, yet complementary, concept to a service.
This is a skeleton of a PEP, presented in order that discussion of
what people want from a “service” has a concrete focus. The
specification is based on work from Trent Nelson.
Note that I'm *not* intending to propose, implement, or champion this
PEP. I present it as is because there is a clear desire to see
something like it, and in the hope that those with more enthusiasm for
it can take it and improve on it to the point where it gets a
champion.
:PEP: XXX
:Title: Background process services
:Version: 0.1
:Last-Modified: 2009-01-30 16:40
:Author: Ben Finney <ben+python(a)benfinney.id.au>
:Status: Draft
:Type: Standards Track
:Content-Type: text/x-rst
:Created: 2009-01-30
:Python-Version: 3.1
:Post-History:
========
Abstract
========
This PEP introduces a package to the Python standard library that
provides an interface to the task of creating a separate background
process, and maintaining an ongoing service channel with that process.
.. contents::
..
Table of Contents:
Abstract
Specification
Interface
``Service`` objects
Motivation
Rationale
Reference Implementation
References
Copyright
=============
Specification
=============
Interface
=========
A new package, `service`, is added to the standard library.
The package defines a class, `Service`, used to set up and maintain
the service.
``Service`` objects
===================
A `Service` instance represents a control channel to a separate,
background process that runs concurrently with the existing program.
The existing program communicates with that background process via the
corresponding `Service` instance.
`__init__(self, func, name, desc=None, startup_type=StartupType.Auto)`
Initialises the parameters of a new service […]
The `func` argument is a callable, which will be called to run in
the background process when the service is started.
[…]
`parse_arguments(args)`
[…]
`is_running()`
Returns ``True`` if the background process is currently running […]
`is_installed()`
Returns ``True`` if […]
`install()`
[…]
`uninstall()`
[…]
`report_event(self, msg, event_type, id, category, data)`
[…]
`exec()`
[…]
`terminate()`
[…]
`request_pause()`
[…]
`request_resume()`
[…]
`send_command(code)`
[…]
`run(args)`
[…]
`start()`
Start a new background process running the `func` callable […]
* On Unix, this creates a new daemon process.
* On MacOS, […].
* On MS Windows, […].
`stop()`
Stop the background process […]
`pause()`
[…]
`resume()`
[…]
==========
Motivation
==========
[…]
=========
Rationale
=========
[…]
========================
Reference Implementation
========================
[…]
==========
References
==========
[…]
=========
Copyright
=========
This work is hereby placed in the public domain. To the extent that
placing a work in the public domain is not legally possible, the
copyright holder hereby grants to all recipients of this work all
rights and freedoms that would otherwise be restricted by copyright.
..
Local variables:
mode: rst
coding: utf-8
time-stamp-start: "^:Last-Modified:[ ]+"
time-stamp-end: "$"
time-stamp-line-limit: 20
time-stamp-format: "%:y-%02m-%02d %02H:%02M"
End:
vim: filetype=rst fileencoding=utf-8 :
On Wed, Jan 28, 2009 at 01:48:46PM +1100, Ben Finney wrote:
> Trent Nelson writes:
>
> > On Tue, Jan 27, 2009 at 08:49:52PM -0500, Jesse Noller wrote:
> > > Windows Services != Unix daemons. Admittedly, I haven't had to do
> > > it in awhile, but from what I remember, windows services are
> > > nowhere near the same type of beast as a unix daemon, and trying
> > > to make a unified daemon library that glosses over the differences
> > > might cause seizures and habitual smoking.
> > >
> > > A unix-only daemon - and then a complimenting windows-only service
> > > library - makes sense. Ideally, they would have close to the same
> > > API, but I don't think log jamming all of the vagaries of both
> > > into one beast makes sense. Then again, my windows-ability is out
> > > of date and rusty.
> >
> > See, I disagree with that as well. Take a look at what Qt did with
> > their QtService class:
> >
> > http://doc.trolltech.com/solutions/qtservice/qtservice.html
>
> Thanks for mentioning that again; I saw another reference to it
> earlier, but didn't respond.
>
> > I'd like to hear arguments for why that QtService interface is
> > perceived as being deficient.
>
> Off the top of my head:
>
> The QtService model seems to be ???start an additional process and
> maintain it at arms length, with a bunch of communication channels
> into and out of it???. Contrariwise, this PEP addresses the use case of
> ???turn the *current* program into a daemon process, so that everything
> further this program does is running inside that process???.
What are you left with at the end of the day? Something you can
start, stop, query status, and log out and have it still run. How
you got there (daemonising the current process) surely isn't as
important as where you ended, especially if it nukes any chance
of cross-platformness?
> It's too big, as a result of the above difference in focus. When
> writing a Unix daemon program, I don't want the great majority of what
> that interface offers.
So I can't persuade you from shifting focus from "When writing a
Unix daemon program, I ...", to "When writing a cross-platform
daemon/service, I ..."?
Sidebar: this definition on the Qt page caught my attention:
"A Windows service or Unix daemon (a "service"),
is a program that runs regardless of whether a
user is logged in or not."
If you're vehemently against the notion of a cross-platform
implementation that utilised Windows services; what about a
Windows implementation that mimicked the Unix behaviour as
best it could? That is, upon daemonisation, the Windows
process would change process group or whatever such that
it would keep running after you log out, until killed.
Not pretty, but it avoids all the problems of Unix-only.
> It misses a lot, again as a result of the mismatch in purpose. After
> setting up a QtService, I need to continue chattering with it to find
> out what's going on inside it, and to do things like change its state
> or log a message. The current PEP, on the other hand, doesn't have the
> model of managing a separate process; the daemon is the *current*
> program, so one can just do things like output or state changes as one
> normally would.
>
> In short: It addresses a different problem to what I'm addressing with
> this PEP.
Indeed it does, but that's 'cause I'm trying to change your focus
from writing a Unix-only daemon to writing a cross-platform service
that just happens to be implemented as a daemon on Unix and as a
service on Windows ;-)
Sure, the interface wouldn't be the same as if it were Unix-only,
but that's what cross-platformness is all about, isn't it?
Trent.
Hello,
Here are some ideas around import. I wish to receive comments and to learn whether they fit python. As I am not a highly experimented programmer, there may be some features or practices I do know know or do not know well. The following ideas are not dependant of each other, but as they all concern the import topic, I thought they could be grouped in a single message anyway.
=== import transmission ===
The module attribute __all__ allows defining names to be exported. I always use it for my own modules, and also sometimes define it in external modules, when absent, or adapt it to my needs. Then I'm happy with a secure "import *".
I find nevertheless an issue in the common case of import "transmission" from module to module, where all imported names usually have to repeated to export. This is not only ugly, imo, but also unpracticle when a single change in an name export list has to be corrected anywhere else; there can easily be a whole chain or even a 'tree' of imports.
I have found a way to deal with that, illustrated in below in a case where module M0 imports M1 and M2, which itself imports M3:
*** M3 ***
M3_names = [actual list of relevant names]
__all__ = ["M3_names"] + M3_names
*** M2 ***
from M3 import *
M2_names = [actual list of relevant /local/ names] + M3_names
__all__ = ["M2_names"] + M2_names
*** M1 ***
M1_names = [actual list of relevant names]
__all__ = ["M1_names"] + M1_names
*** M0 ***
from M1 import *
from M2 import *
M0_names = [actual list of relevant /local/ names] + M1_names + M2_names
__all__ = ["M0_names"] + M0_names
This has the advantage to avoid repetition, and allow a change in a name export list with no need of "chain correction". Still, it is only a personal trick. I wonder whether such a practice could be officialised with a kind of __imported_names__ module attribute, then understood and usable by all:
from Mx import *
__all__ = __imported_names__ + [actual list of relevant /local/ names]
=== semantics of "import *" ===
"import *" is not recommanded for good reasons, as a blind and risky practice. Now, when this is done consciously, kwowing that the imported module defines __all__, then the meaning is totally different, if not opposite. As a consequence, I am not happy at all with the present ambiguity. My proposal would be:
* Either a slightly different syntactic form, meaning "import all names defined for export". [I would like "import **", for '*' and '**' are associated in other cases and this is all about named things. But this point is no issue.] As an additional advantage, imo important, trying this kind of import from a module that does not define __all__ would raise an exception; for instance: ImportError("Module <module name> does not define any name list for export.")
* Or change the semantics of "import *" to require __all__ to be defined in the pointed module. Obviously, this raises compatibility issues that may or may not be adressed using warnings (and/or "from __future__ import import" ;-}) in a change path.
=== package module lookup ===
There is a similar issue at the package/application level: I do not know of any practice or feature that simply allows changing a filename, a directory name, or the organisation of the package's directory tree, without having then to correct all dependance information inside the concerned modules.
I would love a kind of import lookup to be based on the following assumption:
"The main/startup module resides in the package's root dir."
Then any import lookup would automatically explore local, relative, sub directories. E basta!
This would also encourage the good practice of never writing absolute imports -- which is a plague.
There are probably many good ways to implement that feature clearly and efficiently. Here is proposed an example that only aims at clarifying the feature's meaning & usefulness; but does not pretend to be good. The purpose is python to know not only which modules exist, or which modules are intended to be imported, but also where actually they are -- or should be. This does not only address import from client code but also intra-module dependances.
A file called __init__.py is used to identify a package as package and provide information about it. I propose to add a __dir__ attribute to a package, defined inside its __init__.py, that would hold a module directory (in the sense of 'index'). This could be implemented as a python dict holding a set of name:path bindings -- while actually it is pure information. It would be either a complement, or an alternative, to a package's __all__ attribute that lists the module names:
__all__ = ["main", "compute", "userinput"]
__dir__ = ["main":"/", "compute":"/tools", "userinput":"/interface/dialogs"]
Note that the pathes are *relative*. The main point is that __dir__'s value would be set by python itself when it is absent from __init__.py or when it needs beeing updated after any change in the package's tree. So that the process of module lookup would be as follows (provided I properly understand it):
* If current module is not part of a package: usual module lookup is performed.
* If it is part of a package which __dir__ is not yet defined, perform a local tree traversal starting from the module's root dir to reference the modules, then write the result into __dir__. Use it to import.
* If __dir__ is defined, try and import using it. If import fails (maybe the package's organisation has changed), while the module is referenced by __all__, update __dir__ as necessary. Then, try import again.
* When intra-package import finally fails, use PYTHONPATH to try external import, as usual.
Obviously, as long as the package's organisation does not change, a single traversal is necessary for all further imports during all further program executions. Intra-package imports may be faster, even if it is not the primary purpose of the proposal.
Actually, this seems so simple that python aware editors may even provide this automatically: meaning maintain __dir__ inside an __init__.py file. (This is close to the notion of "project" that many editors provide.)
Denis
------
la vida e estranya
When I generate certain files I have to write a date formated as english text,
but the error messages (if there are any) I'd like to keep in the system
language. I propose to add this (or a similar) contextmanager:
@contextmanager
def localectx(category, newlocale=None):
oldlocale = locale.getlocale(category)
locale.setlocale(category, newlocale)
try:
yield
finally:
locale.setlocale(category, oldlocale)
Example usage:
with localectx(locale.LC_TIME,('en','UTF8')):
sdate = time.strftime('%a %b %d %Y')
spec_file.write('''
%%changelog
* %s Maintainer Name <maintainer(a)example.com>
''' % sdate)
-panzi
You should probably at least use a recognized python test suite like
pystones or richards, or use some real world applications for this.
look at the benchmarks for the pypy project for example:
http://tuatara.cs.uni-duesseldorf.de/benchmark.html
Good luck on you endeavor
On Jan 24, 2009, at 2:45 AM, joe wrote:
> Ok, I ran a little test script through CodeAnalyst. basically it is
> just a loop that does member lookups and function calls (this is in
> py2.5 btw, will have to install 3.x and test that too).
>
> The results were kindof interesting. CodeAnalyst is sample-based, so
> I don't know if it's completely reliable. It is usually reasonably
> correct though.
>
> Much of the time was spent in PyEval_EvalFrameEx (which makes sense,
> it being the code that interprets and executes the bytecode). 14633
> samples were spent in it, of which around 2000 were spent in what
> appears to be the switch jump lookup (I'm not sure, I'm not the best
> at reading assembly code). After PyEval_EvalFrameEx, the four next
> most expensive calls were PyWrapper_New (at 2879 samples)
> PyInstance_New (at 1418) PyDict_GetItem (at 1298 samples) and
> PyObject_SetAttr (at 1155).
>
> Here's the (rather hastily thrown together) script:
>
> class A:
> t = 0
>
> class Bleh:
> a = 0
> b = 0
> c = 0
>
> def __init__(self):
> self.a = A()
> self.b = []
> self.c = ""
>
> def doit(self, no=0):
> if no: return
>
> while 1:
> a = self.a
> b = self.b
> c = self.c
> self.a = a
> self.b = b
> self.c = c
> self.doit(1)
> b = Bleh()
> b.doit()
>
> So, it looks like indirect or direct threading could help (if I'm
> reading the results and asm code right). Also the attribute lookups
> did appear to take a significant portion of the time, as did
> PyInstance_New and PyWrapper_New (need to look up what those last two
> do). This is just preliminary though, that isn't a particularly good
> script for profiling, I suspect.
>
> Joe
--
Leonardo Santagada
santagada at gmail.com
On Jan 24, 2009, at 1:53 AM, joe wrote:
> On Fri, Jan 23, 2009 at 2:36 AM, Leonardo Santagada <santagada(a)gmail.com
> > wrote:
>>
>> I don't think TraceMonkey is slower than V8 I believe the last time
>> I looked
>> TraceMonkey was faster than V8 and it is becoming even faster at each
>> interaction.
>>
>> The way TraceMonkey works reminds me a bit of Psyco, although I
>> might be
>> mixing it with the PyPy JIT. But talking about Psyco, why people
>> don't go
>> help Psyco if all they want is a JIT? It is not like the idea of
>> having a
>> JIT on Python is even new... Psyco was optimizing code even before
>> Webkit/V8
>> existed.
>>
>>
>>> This leads me to believe that relatively simple, more general
>>> concepts in
>>> VM design can have a bigger impact then specific, highly
>>> complicated JIT
>>> solutions, in the context of dynamic languages that can't be
>>> easily typed
>>> at compile time.
>>
>> I believe in the exact oposite, and TraceMonkey is probably one of
>> the
>> proofs of that...
>>
> Well, perhaps there's validity to both sides; V8 is still pretty fast.
> I didn't know they'd improved tracemonkey that much, interesting.
> Trace trees struck me as fairly different from psyco's design (it's
> been a while since I looked at the papers for both though); Psyco
> essentially was designed to optimized specific forms of code, while
> trace trees is more generalized.
Yes I confused the design of pypy jit with the psyco one, sorry.
>>> So I've thought of a few ideas for a more (new) streamlined python
>>> VM:
>>>
>>> * Simplify the cpython object model as much as possible, while still
>>> allowing
>>> most of the power of the current model.
>>
>> This would modify the language, so it might be interesting, but would
>> generate something which is not Python.
>
> I don't think so. I didn't mean modify the python object model, I
> meant use a modified version of cpython's implementation of it. The
> cpython C API isn't part of the language standard, after all, and it's
> kindof inefficient and complex, imho.
I never messed much with the cpython implementation of it, but maybe.
If I remember correctly the idea for it is to be reasonably simple so
that it is easy to write extension modules and incorporate it with c
programs.
>>
>>> * Either keep referencing counting, or experiment with some of the
>>> newer
>>> techniques such as pointer escaping. Object models that
>>> exclusively rely
>>> on cyclic GC's have many issues and are hard to get right.
>>
>> Don't know, but a good GC is way faster than what CPython is doing
>> already,
>> but maybe it is a good idea to explore some others perspectives on
>> this.
>>
>
> I disagree. I dislike GCs as a generalized solution, you don't always
> need the overhead of a full GC. Reference counting + a GC seems like
> a better compromise, since there's less work for it to do (or rather,
> the work is spread over time in smaller amounts).
A generational GC does exactly that, it spread the work of a
collection around, not as fine grained as refcounting but maybe
simpler to implement if in the end you also want to get rid of the
GIL... but yes this is still a point that is not a decided matter...
>>> * Possibly modify the bytecode to be register-based, as in
>>> SquirrelFish.
>>> Not sure if this is worth it with python code.
>>
>> Maybe it would help a bit. I don't think it would help more than
>> 10% tops
>> (but I am completely guessing here)
>
> Ah, 10% sounds like it would be worth it, actually. The simple code
> generation SquirrelFish does is interesting too, it essentially
> compiles the opcodes to native code. The code for simpler opcodes
> (and simple execution paths in the more complex ones) are inlined in
> the native code stream, while more complex opcodes are called as
> functions.
There is a tool on pypy to do this also, you should look at it (I
always forget the name, but it would be easy to find on pypy site).
>>> * Use direct threading (which is basically optimizing switch
>>> statements to
>>> be only one or two instructions) for the bytecode loop.
>>
>> The problem with this is (besides the error someone has already
>> stated about
>> your phrasing) that python has really complex bytecodes, so this
>> would also
>> only gain around 10% and it only works with compilers that accept
>> goto
>> labels which the MSVC for example does not (maybe there are more
>> compilers
>> that also doesn't).
>
> Python's bytecode isn't all that complex, when I looked at it. It's
> not that much worse then Squirrelfish's original bytecode
> specification (which I need to look at again, btw, not sure what
> they're doing now). I was kindof surprised, thought it'd be much
> worse.
There was a discussion about this on pypy-dev only a week ago, you
might have some fun looking at the archives.
>>> * Remove string lookups for member access entirely, and replaced
>>> with a
>>> system of unique identifyers. The idea is you would use a hash in
>>> the
>>> types to map a member id to an index. Hashing ints is faster then
>>> strings,
>>> and I've even thought about experimenting with using collapsed
>>> arrays
>>> instead
>>> of hashes. Of course, the design would still need to support string
>>> lookups
>>> when necessary. I've thought about this a lot, and I think you'd
>>> need
>>> the
>>> same general idea as V8's hidden classes for this to work right
>>> (though
>>> instead of classes, it'd just be member/unique id lookup maps).
>>
>> A form of hidden classes is already part of PyPy (but I think that
>> only the
>> jit does this). But you can simply remove string lookups as people
>> can
>> implement special methods to track this on the current Python. As I
>> said
>> before I don't believe changing the semantics of python for the
>> sake of
>> performance is even possible.
>
> Obviously you'd have to go through the effort to *not* change the
> language semantics, which would mean still allowing things like
> __get/setattr__ and __get/setattribute__, a working __dict__, etc.
Thats one of the reason I think that a JIT is probably the only answer
for performance in python. At any time you could appen any of the
special methods to a class...
>>> I'm not sure I'll have the time to anytime soon to prototype these
>>> ideas,
>>> but I
>>> thought I'd kick them out there and see what people say. Note,
>>> I'm in no
>>> way
>>> suggesting any sort of change to the existing cpython VM (it's
>>> way, way
>>> too
>>> early for that kind of talk).
>>
>> If you are not talking about changing CPython VM why not look at
>> Psyco and
>> PyPy? :)
>
> I looked at Psyco. It didn't look like it had much further potential
> to me. It only optimizes certain situations; it's not very
> generalized. PyPy looks interesting though.
Psyco optimizes a lot of situations, the next version I gathered is
going to optimize generator expressions, but yes, pypy I guess is the
long term answer to performance and python.
ps1: I thought you just forgot to send this to the list so I am
replying to it, hope you don't mind.
ps2: I'm not part of pypy core team or anything, so my view is just,
well, my view.
--
Leonardo Santagada
santagada at gmail.com
I have lately been using OpenMP to write parallel C and Fortran code. I
must admit I am impressed. OpenMP is a much better abstraction for
writing concurrent code than using Win32/posix threads directly (or
similarly threading.Thread i Java or Python). What matters most is that
code can be written as sequential, tested, and then parallelised using
compiler pragmas. This is much easier than writing code intended to be
parallel from the start.
Not only is the abstraction more easy to apply, it also leads to fewer
problems with deadlocks, race conditions, livelocks, etc.
I was thinking something similar could be created for Python, e.g. on
top of the existing thread or threading modules, and possibly
multiprocessing. I believe a context manager could be used for this
purpose. What I have in mind is an API that would look approximately
like this (OpenMP pragmas for C on top, proposed Python equivalent below):
#pragma omp parallel
with pymp.Pool() as pool:
#pragma omp for
for item in pool.parallel(<iterable>):
#pragma omp for shedule(guided)
for item in pool.parallel(<iterable>, shed='guided'):
#pragma omp parallel for
with pymp.Pool() as pool:
for item in pool.parallel(<iterable>):
#pragma omp barrier
pool.barrier()
#pragma omp section
pool.section(fun, *args, **kwargs)
#pragma omp parallel sections
with pymp.Pool() as pool:
pool.section(fun1, *args, **kwargs)
pool.section(fun2, *args, **kwargs)
#pragma omp master
if pool.master:
#pragma omp critical
#pragma omp atomic
with pool.lock:
#pragma omp single
with pool.single():
#pragma omp ordered
with pool.ordered():
This is all trivial to program, except for the context manager on top.
It has somehow to get access to the code block below, spawn multiple
threads, and execute that block in each of the threads. I am not sure
how to grab the next executable block as a Python object (so I could
pass it to eval or exec), so a little help would be appreciated :)
Regards,
Sturla Molden
So, I've been kicking around some ideas for an optimized python VM. I freely
admit I'm an amateur at this, but I find the problem of making python code
run faster fascinating. My ideas arose from observing that google V8's JIT
compiler and type system are much simpler compared to TraceMonkey, but is
also faster, and also learning that SquirrelFish is allegedy faster than V8,
even though it doesn't compile to native code at all (which V8 and I
believe TraceMonkey both do).
This leads me to believe that relatively simple, more general concepts in
VM design can have a bigger impact then specific, highly complicated JIT
solutions, in the context of dynamic languages that can't be easily typed
at compile time.
So I've thought of a few ideas for a more (new) streamlined python VM:
* Simplify the cpython object model as much as possible, while still allowing
most of the power of the current model.
* Either keep referencing counting, or experiment with some of the newer
techniques such as pointer escaping. Object models that exclusively rely
on cyclic GC's have many issues and are hard to get right.
* Possibly modify the bytecode to be register-based, as in SquirrelFish.
Not sure if this is worth it with python code.
* Use direct threading (which is basically optimizing switch statements to
be only one or two instructions) for the bytecode loop.
* Remove string lookups for member access entirely, and replaced with a
system of unique identifyers. The idea is you would use a hash in the
types to map a member id to an index. Hashing ints is faster then strings,
and I've even thought about experimenting with using collapsed arrays instead
of hashes. Of course, the design would still need to support string lookups
when necessary. I've thought about this a lot, and I think you'd need the
same general idea as V8's hidden classes for this to work right (though
instead of classes, it'd just be member/unique id lookup maps).
I'm not sure I'll have the time to anytime soon to prototype these ideas, but I
thought I'd kick them out there and see what people say. Note, I'm in no way
suggesting any sort of change to the existing cpython VM (it's way, way too
early for that kind of talk).
references:
v8's design: http://code.google.com/apis/v8/design.html
squirrelfish's design:
http://blog.mozilla.com/dmandelin/2008/06/03/squirrelfish/
Joe
This might be a silly idea, but I was wondering about forcing split()
to return at least X number of items. For example, I might be getting
a comma separated string and i want to split it up into names, but I
might have less than all of them. If it is just 0, 1, or 2, I can use
partition(), but any more and that doesn't work. Besides, I don't care
if the separator is there, just to get the values. Might also make
sense to give the values to give by default.
Example of implementing this:
def split(self, sep=None, max_splits=None, min_items=None):
parts = self.split(sep, max_splits)
if len(parts) < min_items:
parts.extend([None] * (min_items - len(parts)))
return parts
Use would be like this:
a, b, c, d = "1,2,3".split(',', None, 4)
Probably not a great idea, but I'm tossing it out there, anyway.
--
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy
