Prometheus

OliveTin supports basic Prometheus metrics, and the project is interested to hear about what more metrics people would find useful, as well!

To enable Prometheus support;

config.yaml

logLevel: INFO

prometheus:
    enabled: true
    defaultGoMetrics: false

It is required to restart OliveTin after changing the prometheus configuration.

The defaultGoMetrics option will enable the default Go metrics, which expose metrics like go_memstats_alloc_bytes, or go_memstats_heap_alloc_bytes, and generally most people don’t need these.

This will give you metrics available at http://yourserver:1337/metrics. The page should look something like this;

# HELP olivetin_actions_requested_count The actions requested count
# TYPE olivetin_actions_requested_count counter
olivetin_actions_requested_count 0
# HELP olivetin_action_executions_total Total number of finished action executions grouped by result.
# TYPE olivetin_action_executions_total counter
olivetin_action_executions_total{result="success"} 0
olivetin_action_executions_total{result="failed"} 0
olivetin_action_executions_total{result="blocked"} 0
olivetin_action_executions_total{result="timeout"} 0
olivetin_action_executions_total{result="error"} 0
# HELP olivetin_action_execution_duration_seconds Action execution duration in seconds from start to finish.
# TYPE olivetin_action_execution_duration_seconds histogram
olivetin_action_execution_duration_seconds_bucket{le="0.1"} 0
# HELP olivetin_config_action_count Then number of actions in the config file
# TYPE olivetin_config_action_count gauge
olivetin_config_action_count 18
# HELP olivetin_config_reloaded_count The number of times the config has been reloaded
# TYPE olivetin_config_reloaded_count counter
olivetin_config_reloaded_count 1

Failed job monitoring

Finished action executions are counted in olivetin_action_executions_total with a result label:

success

Command finished with exit code 0

failed

Command ran but exited with a non-zero code

timeout

Command exceeded its configured timeout

blocked

Execution was blocked (ACL, rate limit, concurrency, or queue limit)

error

Execution failed before the command ran (for example, invalid arguments)

olivetin_action_execution_duration_seconds records how long each finished execution took.

Example Prometheus alert rules:

groups:
  - name: olivetin
    rules:
      - alert: OliveTinActionFailed
        expr: increase(olivetin_action_executions_total{result="failed"}[15m]) > 0
        labels:
          severity: warning
        annotations:
          summary: OliveTin action failed with non-zero exit code

      - alert: OliveTinActionTimedOut
        expr: increase(olivetin_action_executions_total{result="timeout"}[15m]) > 0
        labels:
          severity: warning
        annotations:
          summary: OliveTin action timed out