从零开始系统学习 Go 编程

Appearance

服务发现 ​

Prometheus 支持多种服务发现机制，自动发现监控目标。

静态配置 ​

最简单的配置方式。

yaml
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
      
  - job_name: 'node'
    static_configs:
      - targets:
          - 'node1:9100'
          - 'node2:9100'
        labels:
          env: 'production'

文件服务发现 ​

从文件读取目标列表。

yaml
scrape_configs:
  - job_name: 'file-sd'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/*.json'
        refresh_interval: 5m

targets.json:

json
[
  {
    "targets": ["web1:8080", "web2:8080"],
    "labels": {
      "job": "web"
    }
  }
]

DNS 服务发现 ​

通过 DNS SRV 记录发现服务。

yaml
scrape_configs:
  - job_name: 'dns-sd'
    dns_sd_configs:
      - names:
          - '_prometheus._tcp.example.com'
        type: 'SRV'
        refresh_interval: 30s

Consul 服务发现 ​

使用 Consul 作为服务发现。

yaml
scrape_configs:
  - job_name: 'consul-sd'
    consul_sd_configs:
      - server: 'localhost:8500'
        services: ['web', 'api']
        tags: ['production']
        refresh_interval: 30s

Kubernetes 服务发现 ​

自动发现 Kubernetes 资源。

yaml
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
            - default
            - monitoring

  - job_name: 'kubernetes-services'
    kubernetes_sd_configs:
      - role: service

  - job_name: 'kubernetes-nodes'
    kubernetes_sd_configs:
      - role: node

Kubernetes 标签示例 ​

yaml
# Pod 注解
metadata:
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '8080'
    prometheus.io/path: '/metrics'

Relabeling 配置 ​

yaml
scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    
    relabel_configs:
      # 只抓取有注解的 Pod
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      
      # 设置端口
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      
      # 设置路径
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

EC2 服务发现 ​

AWS EC2 实例发现。

yaml
scrape_configs:
  - job_name: 'ec2-sd'
    ec2_sd_configs:
      - region: us-east-1
        access_key: 'AKIAXXXX'
        secret_key: 'xxxxx'
        refresh_interval: 60s
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance_name

Docker 服务发现 ​

Docker 容器发现。

yaml
scrape_configs:
  - job_name: 'docker-sd'
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 30s
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container

Relabeling ​

强大的标签重写能力。

relabel_configs（抓取前） ​

yaml
relabel_configs:
  # 保留特定标签
  - source_labels: [env]
    action: keep
    regex: production

  # 删除标签
  - source_labels: [temporary]
    action: labeldrop
    regex: .+

  # 替换标签
  - source_labels: [__address__]
    target_label: instance
    replacement: 'my-$1'

  # 映射标签
  - source_labels: [status]
    target_label: severity
    mapping:
      200: info
      400: warning
      500: critical

metric_relabel_configs（抓取后） ​

yaml
metric_relabel_configs:
  # 只保留特定指标
  - source_labels: [__name__]
    action: keep
    regex: 'http_.*'

  # 删除高基数标签
  - source_labels: [request_id]
    action: labeldrop

最佳实践 ​

  1. 选择合适的发现机制：

    • 静态：固定目标
    • 文件：动态更新目标
    • Kubernetes：云原生环境
    • Consul：服务网格

  2. 合理使用 relabeling：

    • 过滤目标
    • 统一标签
    • 删除无用标签

  3. 控制刷新频率：避免频繁查询

  4. 监控服务发现：

    promql
    prometheus_sd_discovered_targets
prometheus_sd_refresh_failures_total

参考：https://prometheus.io/docs/prometheus/latest/configuration/configuration/