告警规则

Prometheus Alertmanager 提供灵活的告警机制。

告警规则配置

基本结构

groups:
  - name: example
    rules:
      - alert: HighRequestLatency
        expr: job:request_latency_seconds:mean5m > 0.1
        for: 10m
        labels:
          severity: page
        annotations:
          summary: High request latency
          description: Request latency above 100ms (current value: {{ $value }})

规则文件位置

# prometheus.yml
rule_files:
  - "alerts/*.yml"
  - "rules/*.yml"

常用告警规则

HTTP 服务告警

groups:
  - name: http_alerts
    rules:
      # 高错误率
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) 
          / sum(rate(http_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: High HTTP error rate
          description: Error rate is {{ $value | humanizePercentage }}

      # 服务不可用
      - alert: ServiceDown
        expr: up == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: Service {{ $labels.job }} is down
          description: "{{ $labels.instance }} has been down for more than 5 minutes."

      # 高延迟
      - alert: HighLatency
        expr: |
          histogram_quantile(0.95, 
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job)
          ) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: High request latency
          description: P95 latency is {{ $value }}s

系统资源告警

groups:
  - name: system_alerts
    rules:
      # CPU 使用率高
      - alert: HighCPUUsage
        expr: |
          100 - (avg by (instance) 
            (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: High CPU usage on {{ $labels.instance }}
          description: CPU usage is {{ $value | humanize }}%

      # 内存使用率高
      - alert: HighMemoryUsage
        expr: |
          (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
          / node_memory_MemTotal_bytes * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: High memory usage on {{ $labels.instance }}
          description: Memory usage is {{ $value | humanize }}%

      # 磁盘使用率高
      - alert: HighDiskUsage
        expr: |
          (node_filesystem_size_bytes - node_filesystem_free_bytes) 
          / node_filesystem_size_bytes * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: High disk usage on {{ $labels.instance }}
          description: Disk {{ $labels.mountpoint }} usage is {{ $value | humanize }}%

      # 磁盘空间不足
      - alert: DiskSpaceLow
        expr: node_filesystem_avail_bytes / 1024 / 1024 / 1024 < 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: Low disk space
          description: Only {{ $value | humanize }}GB left on {{ $labels.mountpoint }}

应用告警

groups:
  - name: app_alerts
    rules:
      # 实例数量不足
      - alert: TooFewInstances
        expr: count(up{job="myapp"} == 1) < 2
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Too few instances
          description: Only {{ $value }} instances running

      # 请求量异常
      - alert: RequestRateAnomaly
        expr: |
          rate(http_requests_total[5m]) 
          > 2 * rate(http_requests_total[5m] offset 1h)
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Request rate anomaly
          description: Request rate doubled in the last hour

      # 无数据
      - alert: NoData
        expr: absent(up{job="myapp"})
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: No data from myapp
          description: No metrics received for 5 minutes

Alertmanager 配置

基本配置

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alertmanager@example.com'
  smtp_auth_username: 'alertmanager@example.com'
  smtp_auth_password: 'password'

route:
  receiver: 'team-email'
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  group_by: ['alertname', 'severity']

receivers:
  - name: 'team-email'
    email_configs:
      - to: 'team@example.com'
        send_resolved: true

  - name: 'team-slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx'
        channel: '#alerts'

路由配置

route:
  receiver: 'default'
  routes:
    # 严重告警 -> 立即通知
    - match:
        severity: critical
      receiver: 'pagerduty'
      group_wait: 10s
      repeat_interval: 1m

    # 警告 -> 邮件
    - match:
        severity: warning
      receiver: 'team-email'
      group_wait: 30s
      repeat_interval: 4h

    # 按服务分组
    - match_re:
        service: ^(api|web)$
      receiver: 'backend-team'

静默配置

# 创建静默
amtool silence add alertname=HighCPUUsage instance=server01 --duration=1h

# 查看静默
amtool silence query

# 删除静默
amtool silence expire <silence-id>

告警模板

receivers:
  - name: 'team-email'
    email_configs:
      - to: 'team@example.com'
        html: '{{ template "email.html" . }}'

templates:
  - '/etc/alertmanager/templates/*.tmpl'

最佳实践

1. 合理设置 for：避免瞬时抖动
2. 使用标签分组：group_by 按维度分组
3. 设置 severity：区分告警级别
4. 详细的 annotations：包含问题描述和当前值
5. 测试告警：使用 amtool 测试规则

# 测试告警规则
promtool check rules alerts.yml

# 测试告警发送
amtool alert add alertname=TestAlert severity=critical

参考：https://prometheus.io/docs/alerting/latest/configuration/