Supervisor provides a way for a specially written program (which it runs as a subprocess) called an "event listener" to subscribe to "event notifications". An event notification implies that something happened related to a subprocess controlled by supervisord or to supervisord itself. Event notifications are grouped into types in order to make it possible for event listeners to subscribe to a limited subset of event notifications. Supervisor continually emits event notifications as its running even if there are no listeners configured. If a listener is configured and subscribed to an event type that is emitted during supervisord's lifetime, that listener will be notified.

The purpose of the event notification/subscription system is to provide a mechanism for arbitrary code to be run (e.g. send an email, make an HTTP request, etc) when some condition is met. That condition usually has to do with subprocess state. For instance, you may want to notify someone via email when a process crashes and is restarted by Supervisor.

The event notification protocol is based on communication via a subprocess' stdin and stdout. Supervisor sends specially-formatted input to an event listener process' stdin and expects specially-formatted output from an event listener's stdout, forming a request-response cycle. A protocol agreed upon between supervisor and the listener's implementor allows listeners to process event notifications. Event listeners can be written in any language supported by the platform you're using to run Supervisor. Although event listeners may be written in any language, there is special library support for Python in the form of a supervisor.childutils module, which makes creating event listeners in Python slightly easier than in other languages.

A supervisor event listener is specified via a [eventlistener:x] section in the configuration file. Supervisor [eventlistener:x] sections are treated almost exactly like supervisor [program:x] section with the respect to the keys allowed in their configuration except that Supervisor does not respect "capture mode" output from event listener processes (ie. event listeners cannot be PROCESS_COMMUNICATIONS_EVENT event generators). Therefore it is an error to specify stdout_capture_maxbytes or stderr_capture_maxbytes in the configuration of an eventlistener. There is no artificial constraint on the number of eventlistener sections that can be placed into the configuration file.

When an [eventlistener:x] section is defined, it actually defines a "pool", where the number of event listeners in the pool is determined by the numprocs value within the section.

The events parameter of the [eventlistener:x] section specifies the events that will be sent to a listener pool. A well-written event listener will ignore events that it cannot process, but there is no guarantee that a specific event listener won't crash as a result of receiving an event type it cannot handle. Therefore, depending on the listener implementation, it may be important to specify in the configuration that it may receive only certain types of events. The implementor of the event listener is the only person who can tell you what these are (and therefore what value to put in the events configuration). Examples of eventlistener configurations that can be placed in supervisord.conf are as follows.

[eventlistener:memmon]
command=memmon -a 200MB -m bob@example.com
events=TICK_60
      

[eventlistener:mylistener]
command=my_custom_listener.py
events=PROCESS_STATE_CHANGE,TICK_60
      

An advanced feature, specifying an alternate "result handler" for a pool, can be specified via the result_handler parameter of an [eventlistener:x] section in the form of a pkg_resources "entry point" string. The default result handler is supervisord.dispatchers:default_handler. Creating an alternate result handler is not currently documented.

When an event notification is sent by supervisor, all event listener pools which are subscribed to receive events for the event's type (filtered by the events value in the eventlistener section) will be found. One of the listeners in each listener pool will receive the event notification (any "available" listener).

Every process in an event listener pool is treated equally by supervisor. If a process in the pool is unavailable (because it is already processing an event, because it has crashed, or because it has elected to removed itself from the pool), supervisor will choose another process from the pool. If the event cannot be sent because all listeners in the pool are "busy", the event will be buffered and notification will be retried later. "Later" is defined as "the next time that supervisord's select loop executes". For satisfactory event processing performance, you should configure a pool with as many event listener processes as appropriate to handle your event load. This can only be determined empirically for any given workload, there is no "magic number" but to help you determine the optimal number of listeners in a given pool, Supervisor will emit warning messages to its activity log when an event cannot be sent immediately due to pool congestion. There is no artificial constraint placed on the number of processes that can be in a pool, it is limited only by your platform constraints.

A listener pool has an event buffer queue. The queue is sized via the listener pool's buffer_size config file option. If the queue is full and supervisor attempts to buffer an event, supervisor will throw away the oldest event in the buffer and log an error.

An event listener implementation is a program that is willing to accept structured input on its stdin stream and produce structured output on its stdout stream. An event listener implementation should operate in "unbuffered" mode or should flush its stdout every time it needs to communicate back to the supervisord process. Event listeners can be written to be long-running or may exit after a single request (depending on the implementation and the autorestart parameter in the eventlistener's configuration).

An event listener can send arbitrary output to its stderr, which will be logged or ignored by supervisord depending on the stderr-related logfile configuration in its [eventlistener:x] section.

ver:3.0 server:supervisor serial:21 pool:listener poolserial:10 eventname:PROCESS_COMMUNICATION_STDOUT len:54
      

process_name:foo group_name:bar pid:123
This is the data that was sent between the tags
      

import sys

def write_stdout(s):
    sys.stdout.write(s)
    sys.stdout.flush()

def write_stderr(s):
    sys.stderr.write(s)
    sys.stderr.flush()

def main():
    while 1:
        write_stdout('READY\n') # transition from ACKNOWLEDGED to READY
        line = sys.stdin.readline()  # read header line from stdin 
        write_stderr(line) # print it out to stderr
        headers = dict([ x.split(':') for x in line.split() ])
        data = sys.stdin.read(int(headers['len'])) # read the event payload
        write_stderr(data) # print the event payload to stderr
        write_stdout('RESULT 2\nOK') # transition from READY to ACKNOWLEDGED

if __name__ == '__main__':
    main()
    import sys

      

The event types are a controlled set, defined by Supervisor itself. There is no way to add an event type without changing supervisord itself. This is typically not a problem, though, because metadata is attached to events that can be used by event listeners as additional filter criterion, in conjunction with its type.

Event types that may be subscribed to by event listeners are predefined by supervisor and fall into several major categories, including "process state change", "process communication", and "supervisor state change" events. Below are tables describing these event types.

In the below list, we indicate that some event types have a "body" which is a a token set. A token set consists of a set of charaters with space-separated tokens. Each token represents a key-value pair. The key and value are separated by a colon. For example:

processname:cat groupname:cat from_state:STOPPED
    

Token sets do not have a linefeed or carriage return character at their end.

processname:cat groupname:cat from_state:STOPPED tries:0
              

processname:cat groupname:cat from_state:STARTING pid:2766
              

processname:cat groupname:cat from_state:STOPPED tries:0
              

processname:cat groupname:cat from_state:STARTING pid:2766
              

processname:cat groupname:cat from_state:RUNNING expected:0 pid:2766
              

processname:cat groupname:cat from_state:STOPPING pid:2766
              

processname:cat groupname:cat from_state:BACKOFF
              

processname:cat groupname:cat from_state:BACKOFF
              

type:type
data
              

processname:name groupname:name pid:pid
data
              

processname:name groupname:name pid:pid
data
              

processname:name groupname:name pid:pid
data
              

processname:name groupname:name pid:pid
data
              

when:1201063880