高可用部署

Grafana 支持高可用部署模式，通过数据库后端和负载均衡实现高可用。

高可用架构

架构设计

Load Balancer (Nginx/HAProxy)
       │
       ├─ Grafana Instance 1
       │     └─ Session Store: Database
       │     └─ Dashboard Store: Database
       │
       ├─ Grafana Instance 2
       │     └─ Session Store: Database
       │     └─ Dashboard Store: Database
       │
       └─ Grafana Instance 3
             └─ Session Store: Database
             └─ Dashboard Store: Database
       
       ↓
   Shared Database (PostgreSQL/MySQL)
       │
       ├─ Dashboards
       ├─ Users
       ├─ Annotations
       └─ Sessions

数据库配置

PostgreSQL 配置

yaml

# grafana.ini
[database]
type = postgres
host = postgres:5432
name = grafana
user = grafana
password = password

[session]
provider = postgres
provider_config = postgres://grafana:password@postgres:5432/grafana
cookie_secure = true
cookie_name = grafana_sess

MySQL 配置

yaml

# grafana.ini
[database]
type = mysql
host = mysql:3306
name = grafana
user = grafana
password = password
ssl_mode = disable

[session]
provider = mysql
provider_config = grafana:password@tcp(mysql:3306)/grafana
cookie_secure = true

数据库初始化

sql

-- PostgreSQL
CREATE DATABASE grafana;
CREATE USER grafana WITH PASSWORD 'password';
GRANT ALL PRIVILEGES ON DATABASE grafana TO grafana;

-- MySQL
CREATE DATABASE grafana CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'grafana'@'%' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON grafana.* TO 'grafana'@'%';
FLUSH PRIVILEGES;

Docker Compose 部署

docker-compose.yml

yaml

version: '3.8'

services:
  grafana:
    image: grafana/grafana:10.4.0
    deploy:
      replicas: 3
      update_config:
        parallelism: 1
        delay: 10s
      restart_policy:
        condition: on-failure
        max_attempts: 3
    ports:
      - "3000:3000"
    environment:
      - GF_DATABASE_TYPE=postgres
      - GF_DATABASE_HOST=postgres:5432
      - GF_DATABASE_NAME=grafana
      - GF_DATABASE_USER=grafana
      - GF_DATABASE_PASSWORD=password
      - GF_SESSION_PROVIDER=postgres
      - GF_SESSION_PROVIDER_CONFIG=postgres://grafana:password@postgres:5432/grafana
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_INSTALL_PLUGINS=grafana-clock-panel
    depends_on:
      - postgres
    volumes:
      - grafana-config:/etc/grafana
      - grafana-plugins:/var/lib/grafana/plugins

  postgres:
    image: postgres:16.2
    environment:
      - POSTGRES_USER=grafana
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=grafana
    volumes:
      - postgres-data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U grafana"]
      interval: 10s
      timeout: 5s
      retries: 5

  nginx:
    image: nginx:1.26.0
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    depends_on:
      - grafana

volumes:
  grafana-config:
  grafana-plugins:
  postgres-data:

nginx.conf

nginx

upstream grafana {
    least_conn;
    server grafana1:3000 max_fails=3 fail_timeout=30s;
    server grafana2:3000 max_fails=3 fail_timeout=30s;
    server grafana3:3000 max_fails=3 fail_timeout=30s;
}

server {
    listen 80;
    
    location / {
        proxy_pass http://grafana;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        
        proxy_connect_timeout 10s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
    }
    
    location /api/live {
        proxy_pass http://grafana;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Kubernetes 部署

Deployment

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  replicas: 3
  selector:
    matchLabels:
      app: grafana
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:10.4.0
        ports:
        - containerPort: 3000
        env:
        - name: GF_DATABASE_TYPE
          value: "postgres"
        - name: GF_DATABASE_HOST
          valueFrom:
            configMapKeyRef:
              name: grafana-config
              key: database-host
        - name: GF_DATABASE_USER
          valueFrom:
            secretKeyRef:
              name: grafana-secret
              key: database-user
        - name: GF_DATABASE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-secret
              key: database-password
        resources:
          requests:
            memory: "256Mi"
            cpu: "500m"
          limits:
            memory: "512Mi"
            cpu: "1000m"
        livenessProbe:
          httpGet:
            path: /api/health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /api/health
            port: 3000
          initialDelaySeconds: 5
          periodSeconds: 5
        volumeMounts:
        - name: grafana-storage
          mountPath: /var/lib/grafana
      volumes:
      - name: grafana-storage
        persistentVolumeClaim:
          claimName: grafana-pvc

Service

yaml

apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 3000
  selector:
    app: grafana

ConfigMap

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-config
data:
  database-host: "postgres:5432"
  grafana.ini: |
    [database]
    type = postgres
    host = postgres:5432
    name = grafana
    
    [session]
    provider = postgres
    
    [server]
    root_url = http://grafana.example.com
    
    [security]
    admin_user = admin

Secret

yaml

apiVersion: v1
kind: Secret
metadata:
  name: grafana-secret
type: Opaque
data:
  database-user: Z3JhZmFuYQ==
  database-password: cGFzc3dvcmQ=
  admin-password: YWRtaW4=

配置同步

数据源 Provisioning

yaml

# /etc/grafana/provisioning/datasources/datasources.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    isDefault: true
    
  - name: Loki
    type: loki
    url: http://loki:3100

仪表板 Provisioning

yaml

# /etc/grafana/provisioning/dashboards/dashboards.yml
apiVersion: 1

providers:
  - name: 'default'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    options:
      path: /var/lib/grafana/dashboards

告警 Provisioning

yaml

# /etc/grafana/provisioning/alerting/alerting.yml
apiVersion: 1

policies:
  - orgId: 1
    receiver: default-receiver
    
contactPoints:
  - orgId: 1
    name: default-receiver
    receivers:
      - type: slack
        settings:
          url: https://hooks.slack.com/services/xxx

健康检查

健康检查 API

bash

# 基础健康检查
curl http://localhost:3000/api/health

# 详细健康检查
curl http://localhost:3000/api/health/check

响应示例

json

{
  "commit": "abc123",
  "database": "ok",
  "version": "10.4.0"
}

监控 Grafana

Prometheus 指标

yaml

# grafana.ini
[metrics]
enabled = true
interval_seconds = 10

[metrics.graphite]
address = localhost:2003
prefix = grafana

内置指标

promql

# Grafana API 请求延迟
grafana_api_response_time_seconds{method="GET"}

# Grafana 活跃会话
grafana_active_sessions_count

# Grafana 数据源查询数量
grafana_datasource_query_total{datasource="prometheus"}

# Grafana 仪表板数量
grafana_dashboards_count

备份和恢复

数据库备份

bash

# PostgreSQL 备份
pg_dump -U grafana grafana > grafana_backup.sql

# PostgreSQL 恢复
psql -U grafana grafana < grafana_backup.sql

# MySQL 备份
mysqldump -u grafana -p grafana > grafana_backup.sql

# MySQL 恢复
mysql -u grafana -p grafana < grafana_backup.sql

仪表板导出

bash

# 导出所有仪表板
curl -H "Authorization: Bearer xxx" \
  http://localhost:3000/api/search?type=dash-db | \
  jq -r '.[] | .uid' | \
  xargs -I {} curl -H "Authorization: Bearer xxx" \
  http://localhost:3000/api/dashboards/uid/{}

故障恢复

自动恢复策略

yaml

# Kubernetes Pod Disruption Budget
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: grafana-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: grafana

故障排查

bash

# 检查 Grafana 状态
docker logs grafana
kubectl logs grafana-xxx

# 检查数据库连接
curl http://localhost:3000/api/health

# 检查会话状态
curl http://localhost:3000/api/user

最佳实践

数据库后端：使用 PostgreSQL 或 MySQL 存储状态
会话共享：配置共享会话存储
负载均衡：使用 Nginx 或 HAProxy 分发请求
健康检查：配置定期健康检查和自动恢复
数据备份：定期备份数据库和仪表板
配置同步：使用 Provisioning 自动配置
监控告警：监控 Grafana 自身状态
滚动更新：使用滚动更新策略减少停机时间

参考：https://grafana.com/docs/grafana/latest/setup-grafana/set-up-grafana/

高可用部署 ​

高可用架构 ​

架构设计 ​

数据库配置 ​

PostgreSQL 配置 ​

MySQL 配置 ​

数据库初始化 ​

Docker Compose 部署 ​

docker-compose.yml ​

nginx.conf ​

Kubernetes 部署 ​

Deployment ​

Service ​

ConfigMap ​

Secret ​

配置同步 ​

数据源 Provisioning ​

仪表板 Provisioning ​

告警 Provisioning ​

健康检查 ​

健康检查 API ​

响应示例 ​

监控 Grafana ​

Prometheus 指标 ​

内置指标 ​

备份和恢复 ​

数据库备份 ​

仪表板导出 ​

故障恢复 ​

自动恢复策略 ​

故障排查 ​

最佳实践 ​

高可用部署

高可用架构

架构设计

数据库配置

PostgreSQL 配置

MySQL 配置

数据库初始化

Docker Compose 部署

docker-compose.yml

nginx.conf

Kubernetes 部署

Deployment

Service

ConfigMap

Secret

配置同步

数据源 Provisioning

仪表板 Provisioning

告警 Provisioning

健康检查

健康检查 API

响应示例

监控 Grafana

Prometheus 指标

内置指标

备份和恢复

数据库备份

仪表板导出

故障恢复

自动恢复策略

故障排查

最佳实践