AMQP listening and user-facing daemon
c.justin88 at gmail.com
Sun Jul 7 23:57:55 CEST 2013
What is the best approach to writing a concurrent daemon that can
execute callbacks for different types of events (AMQP messages, parsed
output of a subprocess, HTTP requests)?
I am considering [twisted], the built-in [threading] module, and
[greenlet]. I must admit that I am very unfamiliar with concurrent
programming and Python programming in general (formerly a data analysis
driven procedural programmer). Any resources on threaded/concurrent
programming (specifically daemons...not just multi-threading a single
task) would be much appreciated.
1) Listens into AMQP messaging queues and executes callbacks when
Example: Immediately after startup, the daemon continuously listens to
the [Openstack Notifications messaging queue]. When a virtual machine
is launched, a notification is generated by Openstack with the hostname,
IP address, etc. The daemon should read this message and write some info
to a log (or POST the info to a server, or notify the user...something
2) Parse the output of a subprocess and execute callbacks based on the
Example: Every 30 seconds, a system command "[qstat]" is run to query
a job resource manager (e.g. TORQUE). Similar callbacks to 1).
3) Receive requests from a user and process them. I think this will be
via WSGI HTTP.
Example: User submits an XML template with virtual machine templates.
The daemon does some simple XML parsing and writes a job script for the
job resource manager. The job is submitted to the resource manager and
the daemon continually checks for the status of the job with "qstat" and
for messages from AMQP. It should return "live" feedback to the user and
write to a log.
Justin Chiu TRIUMF
More information about the Python-list