Setting Up Grafana OnCall for Incident Management

Getting grafana-oncall right from the start saves hours of debugging later. In this comprehensive guide, we'll cover everything from initial setup to production-ready configuration, including incidents and alerts considerations.

Prerequisites

• A registered domain name (for public-facing services)
• A VPS running Ubuntu 22.04 or later (2GB+ RAM recommended)
• A reverse proxy configured (Nginx or Traefik)
• Basic familiarity with the Linux command line

Docker Compose Setup

When scaling this setup, consider vertical scaling (adding more RAM/CPU) first, as it's simpler to implement. Horizontal scaling adds complexity but may be necessary for high-traffic applications.

# docker-compose.yml
version: '3.8'
services:
  grafana-oncall:
    image: grafana-oncall/grafana-oncall:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - grafana-oncall_data:/data
      - grafana-oncall_config:/config
    environment:
      - TZ=UTC
      - PUID=1000
      - PGID=1000
    depends_on:
      - db

  db:
    image: postgres:16-alpine
    restart: unless-stopped
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=grafana-oncall
      - POSTGRES_USER=grafana-oncall
      - POSTGRES_PASSWORD=changeme

volumes:
  grafana-oncall_data:
  grafana-oncall_config:
  db_data:

This configuration provides a good balance between performance and resource usage. For high-traffic scenarios, you may need to increase the limits further.

• Set up fail2ban for brute force protection
• Use SSH keys instead of password authentication
• Enable firewall and allow only necessary ports
• Keep all software components up to date
• Use strong, unique passwords for all services

Initial Configuration

The incidents component plays a crucial role in the overall architecture. Understanding how it interacts with grafana-oncall will help you make better configuration decisions.

# Reverse proxy configuration
server {
    listen 443 ssl http2;
    server_name grafana-oncall.example.com;

    ssl_certificate /etc/letsencrypt/live/grafana-oncall.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/grafana-oncall.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_buffering off;
        client_max_body_size 0;
    }
}

Each line in the configuration serves a specific purpose. The comments explain the reasoning behind each setting, making it easier to customize for your specific use case.

Security Implications

The incidents component plays a crucial role in the overall architecture. Understanding how it interacts with grafana-oncall will help you make better configuration decisions.

Reverse Proxy Integration

Security should be a primary consideration when configuring grafana-oncall. Always use strong passwords, keep software updated, and restrict network access to only the necessary ports and IP addresses.

# docker-compose.yml
version: '3.8'
services:
  grafana-oncall:
    image: grafana-oncall/grafana-oncall:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    volumes:
      - grafana-oncall_data:/data
      - grafana-oncall_config:/config
    environment:
      - TZ=UTC
      - PUID=1000
      - PGID=1000
    depends_on:
      - db

  db:
    image: postgres:16-alpine
    restart: unless-stopped
    volumes:
      - db_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=grafana-oncall
      - POSTGRES_USER=grafana-oncall
      - POSTGRES_PASSWORD=changeme

volumes:
  grafana-oncall_data:
  grafana-oncall_config:
  db_data:

Note that file paths may vary depending on your Linux distribution. The examples here are for Debian/Ubuntu; adjust paths accordingly for RHEL/CentOS-based systems.

Summary

You've successfully configured grafana-oncall on your VPS. Remember to monitor performance, keep your software updated, and maintain regular backups. If you run into issues, consult the official documentation or open a support ticket for assistance.