告警配置

Grafana 提供强大的告警系统，支持多种告警规则和通知渠道。

告警架构

Grafana Alerting
       │
       ├─ Alert Rules
       │     └─ 条件评估
       │
       ├─ Contact Points
       │     └─ 通知渠道
       │
       └─ Notification Policies
             └─ 路由规则

告警规则

创建告警规则

通过界面创建

导航到 Alerting → Alert rules
点击 New alert rule
配置规则参数

规则配置示例

yaml

name: High Error Rate
condition: WHEN avg() OF query(A) IS ABOVE 5
evaluate: every 1m
for: 5m
annotations:
  summary: "Error rate is {{ $value }}%"
  description: "Service {{ $labels.service }} error rate exceeds threshold"
labels:
  severity: critical
  team: backend

Prometheus 查询告警

yaml

# 基础告警规则
- name: ServiceDown
  condition: WHEN last() OF query(A) IS BELOW 1
  query:
    A: up{job="app"}
  annotations:
    summary: "Service {{ $labels.instance }} is down"

# 错误率告警
- name: HighErrorRate
  condition: WHEN avg() OF query(A) IS ABOVE 5
  query:
    A: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
  labels:
    severity: warning

# CPU 告警
- name: HighCPUUsage
  condition: WHEN avg() OF query(A) IS ABOVE 80
  query:
    A: 100 - avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100
  labels:
    severity: warning

多条件告警

yaml

name: CriticalErrorRate
condition: 
  - WHEN avg() OF query(A) IS ABOVE 10 FOR 2m
  - AND WHEN avg() OF query(B) IS ABOVE 100
query:
  A: sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
  B: sum(rate(http_requests_total[5m]))
labels:
  severity: critical

告警表达式

promql

# 简单表达式
rate(http_requests_total[5m]) > 100

# 组合表达式
sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) > 10 
AND 
sum(rate(http_requests_total[5m])) by (service) > 1000

# 数学运算表达式
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 90

通知渠道

Email 通知

yaml

# Email 配置
type: email
settings:
  addresses: team@example.com
  singleEmail: false

Slack 通知

yaml

# Slack 配置
type: slack
settings:
  url: https://hooks.slack.com/services/xxx
  channel: alerts
  username: grafana
  icon_emoji: :warning:

PagerDuty 通知

yaml

# PagerDuty 配置
type: pagerduty
settings:
  serviceKey: xxxxxx
  severity: critical

Webhook 通知

yaml

# Webhook 配置
type: webhook
settings:
  url: http://webhook-server/alert
  httpMethod: POST
  basicAuthUser: user
  basicAuthPassword: password

Telegram 通知

yaml

# Telegram 配置
type: telegram
settings:
  bottoken: xxxxxx
  chatid: -123456789

企业微信通知

yaml

# 企业微信配置
type: wechat
settings:
  api_url: https://qyapi.weixin.qq.com/cgi-bin/message/send?access_token=xxx
  agent_id: 1000002
  corp_id: xxx
  secret: xxx
  to_user: "@all"

Contact Points（联系点）

配置 Contact Points

yaml

apiVersion: 1

contactPoints:
  - orgId: 1
    name: backend-team
    receivers:
      - uid: email-receiver
        type: email
        settings:
          addresses: backend@example.com
      - uid: slack-receiver
        type: slack
        settings:
          url: https://hooks.slack.com/services/xxx
          channel: backend-alerts

联系点类型

类型	说明	使用场景
Email	邮件通知	低优先级告警
Slack	Slack 消息	团队协作
PagerDuty	电话/短信	高优先级告警
Webhook	自定义 HTTP	集成第三方系统
Telegram	Telegram 消息	国际团队
OpsGenie	运维管理平台	企业运维

Notification Policies（通知策略）

配置路由规则

yaml

apiVersion: 1

policies:
  - orgId: 1
    receiver: default-receiver
    group_by:
      - grafana_folder
      - alertname
    routes:
      - receiver: backend-team
        match:
          team: backend
        continue: false
      - receiver: frontend-team
        match:
          team: frontend
        continue: false
      - receiver: critical-alerts
        match:
          severity: critical
        group_wait: 10s
        group_interval: 1m
        repeat_interval: 5m

路由参数说明

参数	说明	默认值
group_by	分组标签	grafana_folder, alertname
group_wait	首次等待时间	30s
group_interval	分组间隔	5m
repeat_interval	重复发送间隔	4h

告警模板

自定义消息模板

yaml

apiVersion: 1

templates:
  - orgId: 1
    name: custom_template
    template: |
      {{ define "custom.alert" }}
      Alert: {{ .Labels.alertname }}
      Summary: {{ .Annotations.summary }}
      Value: {{ .Value }}
      Labels:
      {{ range .Labels.SortedPairs }}
        {{ .Name }}: {{ .Value }}
      {{ end }}
      {{ end }}

使用模板

yaml

contactPoints:
  - name: custom-alerts
    receivers:
      - type: email
        settings:
          addresses: team@example.com
          subject: '{{ template "custom.alert" . }}'
          body: |
            {{ template "custom.alert" . }}
            
            Details:
            {{ .Annotations.description }}

告警抑制

配置抑制规则

yaml

apiVersion: 1

muteTimings:
  - orgId: 1
    name: maintenance-window
    time_intervals:
      - start_time: "22:00"
        end_time: "06:00"
        location: "Asia/Shanghai"

抑制配置

yaml

rules:
  - orgId: 1
    name: inhibit-during-maintenance
    sourceMatchers:
      - alertname = MaintenanceMode
    targetMatchers:
      - alertname != MaintenanceMode
    equal:
      - service

告警状态

状态流转

Normal → Pending → Firing → Resolved → Normal

状态说明

状态	说明
Normal	正常状态，未触发
Pending	待触发，等待持续时间
Firing	已触发，发送告警
Resolved	已恢复，发送恢复通知

状态持续时间

yaml

evaluate_every: 1m
for: 5m  # 持续5分钟后触发

告警 API

创建告警规则

bash

curl -X POST http://admin:admin@localhost:3000/api/v1/provisioning/alert-rules \
  -H "Content-Type: application/json" \
  -d '{
    "uid": "alert-uid",
    "title": "High Error Rate",
    "condition": "B",
    "data": [
      {
        "refId": "A",
        "queryType": "",
        "relativeTimeRange": {
          "from": 600,
          "to": 0
        },
        "datasourceUid": "prometheus-uid",
        "model": {
          "expr": "rate(http_requests_total[5m])",
          "refId": "A"
        }
      }
    ],
    "noDataState": "NoData",
    "execErrState": "Alerting",
    "for": "5m"
  }'

查询告警状态

bash

curl -X GET http://admin:admin@localhost:3000/api/v1/alerts

暂停告警

bash

curl -X POST http://admin:admin@localhost:3000/api/alerts/1/pause

常用告警规则示例

应用监控告警

yaml

# 服务可用性
- name: ServiceDown
  condition: up{job="app"} == 0
  severity: critical
  
# 错误率告警
- name: HighErrorRate
  condition: 
    sum(rate(http_requests_total{status=~"5.."}[5m])) 
    / sum(rate(http_requests_total[5m])) * 100 > 5
  severity: warning

# 响应时间告警  
- name: HighLatency
  condition:
    histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
  severity: warning

资源监控告警

yaml

# CPU 告警
- name: HighCPUUsage
  condition: 
    100 - avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100 > 80
  severity: warning

# 内存告警
- name: HighMemoryUsage
  condition:
    (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) 
    / node_memory_MemTotal_bytes * 100 > 90
  severity: warning

# 磁盘告警
- name: HighDiskUsage
  condition:
    (node_filesystem_size_bytes - node_filesystem_free_bytes)
    / node_filesystem_size_bytes * 100 > 85
  severity: warning

容器监控告警

yaml

# Pod 状态告警
- name: PodNotReady
  condition: kube_pod_status_ready{condition="true"} == 0
  severity: warning

# 容器重启告警
- name: ContainerRestart
  condition: increase(kube_pod_container_status_restarts_total[1h]) > 5
  severity: warning

# 资源限制告警
- name: ContainerCPUThrottling
  condition: rate(container_cpu_cfs_throttled_seconds_total[5m]) > 0.5
  severity: warning

最佳实践

分级告警：使用 severity 标签区分告警级别
合理阈值：根据业务情况设置合理的告警阈值
避免告警风暴：设置合适的 group_wait 和 repeat_interval
使用抑制：在维护窗口期抑制相关告警
添加上下文：在 annotations 中添加有用的信息
测试告警：定期测试告警通知渠道是否正常
文档化：记录每个告警规则的业务含义
定期审查：定期审查和优化告警规则

参考：https://grafana.com/docs/grafana/latest/alerting/

告警配置 ​

告警架构 ​

告警规则 ​

创建告警规则 ​

通过界面创建 ​

规则配置示例 ​

Prometheus 查询告警 ​

多条件告警 ​

告警表达式 ​

通知渠道 ​

Email 通知 ​

Slack 通知 ​

PagerDuty 通知 ​

Webhook 通知 ​

Telegram 通知 ​

企业微信通知 ​

Contact Points（联系点） ​

配置 Contact Points ​

联系点类型 ​

Notification Policies（通知策略） ​

配置路由规则 ​

路由参数说明 ​

告警模板 ​

自定义消息模板 ​

使用模板 ​

告警抑制 ​

配置抑制规则 ​

抑制配置 ​

告警状态 ​

状态流转 ​

状态说明 ​

状态持续时间 ​

告警 API ​

创建告警规则 ​

查询告警状态 ​

暂停告警 ​

常用告警规则示例 ​

应用监控告警 ​

资源监控告警 ​

容器监控告警 ​

最佳实践 ​

告警配置

告警架构

告警规则

创建告警规则

通过界面创建

规则配置示例

Prometheus 查询告警

多条件告警

告警表达式

通知渠道

Email 通知

Slack 通知

PagerDuty 通知

Webhook 通知

Telegram 通知

企业微信通知

Contact Points（联系点）

配置 Contact Points

联系点类型

Notification Policies（通知策略）

配置路由规则

路由参数说明

告警模板

自定义消息模板

使用模板

告警抑制

配置抑制规则

抑制配置

告警状态

状态流转

状态说明

状态持续时间

告警 API

创建告警规则

查询告警状态

暂停告警

常用告警规则示例

应用监控告警

资源监控告警

容器监控告警

最佳实践