集成环境部署

Prometheus + Grafana + SLA Exporter

207:Prometheus :9092、Grafana :3001、SLA Exporter :8781 的部署与配置。

集成环境部署:Prometheus + Grafana + SLA Exporter

本文定位:207 上 SLA 监控集成环境 的部署与配置教程(独立类目)。作业 8 讲「大盘长什么样、指标怎么算」;本文讲「三个服务怎么装、端口为什么这样分、配置文件写什么」。

环境:192.168.110.207

- Prometheus :9092

- Grafana :3001(账号 admin / admin123)

- SLA Exporter :8781

Langfuse 仍占用 :3000,MinIO API :9090、MinIO Console 127.0.0.1:9091,不要抢。

HTML:docs/html_cursor/集成环境部署_Prometheus_Grafana_SLA_cursor.html

大盘成品(作业 8):http://192.168.110.207:3001/d/ai-platform-sla

作业正文:作业8:SLA 监控大盘


目录

  1. 207 端口与角色
  2. 架构
  3. 目录与配置文件
  4. 一键部署
  5. 手工部署(207 本机)
  6. 验收
  7. 日常运维
  8. 完整配置

207 端口与角色

本机已经有一批服务。监控栈必须错开端口,否则 Docker 绑定失败(实战踩过::9091 已被 MinIO Console 占用)。

服务地址谁占用本教程
Difyhttp://192.168.110.207(80)已有只探测,不改
Langfuse:3000已有只探测,不改
MinIO API:9090已有不是 Prometheus
MinIO Console127.0.0.1:9091已有不要再绑 9091
Xinference:9997已有只探测
作业5 健康 API:8779已有只探测
作业7 自愈 API:8780已有不改
SLA Exporter:8781本环境Python 进程,暴露 /metrics
Grafana:3001本环境容器 hw8-grafana
Prometheus:9092本环境容器 hw8-prometheus(容器内仍是 9090)

_config.py 对应常量:

常量值
SLA_EXPORTER_PORT / SLA_EXPORTER_BASE8781 / http://192.168.110.207:8781
HOMEWORK8_PROMETHEUS_PORT / _URL9092 / http://192.168.110.207:9092
HOMEWORK8_GRAFANA_PORT / _URL3001 / http://192.168.110.207:3001
HOMEWORK8_GRAFANA_USER / _PASSadmin / admin123
HOMEWORK8_DASHBOARD_UIDai-platform-sla
HOMEWORK8_PATCH_DIR/opt/workbuddy/patches_cursor/作业8_SLA监控大盘
HOMEWORK8_DATA_DIR/opt/workbuddy/data/homework8

Grafana 直达大盘:http://192.168.110.207:3001/d/ai-platform-sla


架构

text
207 宿主机
├─ python3 作业8_sla_exporter.py     0.0.0.0:8781
│     /health  /metrics
│     每 60s 健康检查 + Langfuse P99 + 演示项
│
├─ Docker  hw8-prometheus            0.0.0.0:9092 → 容器:9090
│     scrape host.docker.internal:8781  每 15s
│
└─ Docker  hw8-grafana               0.0.0.0:3001 → 容器:3000
      数据源 http://prometheus:9090(compose 内网名)
      预置大盘 uid=ai-platform-sla

Prometheus 在容器里访问宿主机上的 Exporter,靠 extra_hosts: host.docker.internal:host-gateway。 Grafana 访问 Prometheus 用 compose 服务名 prometheus:9090,不要写 192.168.110.207:9092(容器内走 overlay 网更快、也更稳)。


目录与配置文件

全部挂在作业 8 子目录(程序与文档同步约定:按文档名分子目录)。

text
patches_cursor/作业8_SLA监控大盘/
├── docker-compose.monitoring.yml
├── prometheus/prometheus.yml
└── grafana/
    ├── dashboards/ai-platform-sla.json
    └── provisioning/
        ├── datasources/prometheus.yml
        └── dashboards/dashboards.yml

scripts_bash_cursor/作业8_SLA监控大盘/
├── 作业8_01_部署监控栈.sh      # docker compose up
└── 作业8_启动_exporter.sh     # nohup python3 :8781

scripts_cursor/作业8_SLA监控大盘/
├── 作业8_01_部署监控栈.py      # Windows SSH 调上面的 sh
├── 作业8_02_启动_exporter.py
├── 作业8_sla_exporter.py
└── 作业8_sla_metrics.py

一键部署

Windows 项目根(先同步再部署):

bash
python scripts_cursor/00_部署_workbuddy到207.py
python scripts_cursor/作业8_SLA监控大盘/作业8_01_部署监控栈.py
python scripts_cursor/作业8_SLA监控大盘/作业8_02_启动_exporter.py
脚本干什么成功标志
作业8_01_部署监控栈.pySSH 207,docker compose up -d --force-recreate 起 hw8-prometheus / hw8-grafana[OK] 监控栈 http://192.168.110.207:9092 / http://192.168.110.207:3001
作业8_02_启动_exporter.pyFORCE_RESTART=1 拉起 :8781[OK] SLA Exporter http://192.168.110.207:8781/metrics

前置:207 已装 Docker,能 docker compose(或 docker-compose);/opt/workbuddy 已同步。


手工部署(207 本机)

4.1 起 Prometheus + Grafana

bash
cd /opt/workbuddy/patches_cursor/作业8_SLA监控大盘
docker compose -f docker-compose.monitoring.yml up -d --force-recreate
curl -sf http://127.0.0.1:9092/-/healthy
curl -sf http://127.0.0.1:3001/api/health

期望:Prometheus Server is Healthy. 以及 Grafana JSON 里 "database": "ok"。

4.2 起 SLA Exporter

bash
FORCE_RESTART=1 bash /opt/workbuddy/scripts_bash_cursor/作业8_SLA监控大盘/作业8_启动_exporter.sh

进程、日志:

项路径
PID/tmp/hw8_sla_exporter.pid
日志/tmp/hw8_sla_exporter.log
状态文件/opt/workbuddy/data/homework8/sla_state.json

改了 Python 必须 FORCE_RESTART=1,否则还是旧进程。


验收

bash
curl -sf http://192.168.110.207:8781/health
curl -sf http://192.168.110.207:8781/metrics | grep -E 'ai_platform_component_up|name='
curl -sf http://192.168.110.207:9092/api/v1/targets
curl -sf http://192.168.110.207:3001/api/health

Prometheus Targets 里 job ai-platform-sla-exporter 应为 UP。 浏览器:http://192.168.110.207:3001/d/ai-platform-sla(admin / admin123)。 组件健康一行必须能看出是 Dify / Langfuse / vLLM / Xinference / 健康 API,不能只显示 UP。

Windows 再跑:

bash
python scripts_cursor/作业8_SLA监控大盘/作业8_04_验收大盘.py

成功:[PASS] 作业8 SLA 大盘验收。


日常运维

动作命令
看容器docker ps --filter name=hw8-
重启 Grafana(改过大盘 JSON 后)docker restart hw8-grafana
重建监控栈重跑 作业8_01_部署监控栈.py
重启 ExporterFORCE_RESTART=1 bash .../作业8_启动_exporter.sh
停监控栈cd /opt/workbuddy/patches_cursor/作业8_SLA监控大盘 && docker compose -f docker-compose.monitoring.yml down
停 Exporterkill $(cat /tmp/hw8_sla_exporter.pid)

典型失败

现象原因处理
listen tcp 0.0.0.0:9091: address already in useMinIO Console 占了 9091本环境 Prometheus 必须用 9092,不要改回 9091
Grafana 容器 read-only file system 挂载失败把 dashboard JSON 挂到另一个 :ro 目录的子路径大盘挂在 /etc/grafana/dashboards,provisioning 只挂 yaml
Prometheus Target DOWNExporter 没起,或容器访问不到宿主机 8781先 curl 127.0.0.1:8781/metrics;确认 compose 有 host.docker.internal:host-gateway
大盘还是旧布局Grafana 没加载新 JSON同步后再 docker restart hw8-grafana,浏览器 Ctrl+F5
Exporter 指标不变旧 python 进程没杀FORCE_RESTART=1 重拉

完整配置

docker-compose.monitoring.yml

yaml
services:
  prometheus:
    image: prom/prometheus:v2.54.1
    container_name: hw8-prometheus
    restart: unless-stopped
    extra_hosts:
      - "host.docker.internal:host-gateway"
    ports:
      - "9092:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=15d"

  grafana:
    image: grafana/grafana:11.2.0
    container_name: hw8-grafana
    restart: unless-stopped
    depends_on:
      - prometheus
    ports:
      - "3001:3000"
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: admin123
      GF_USERS_ALLOW_SIGN_UP: "false"
      GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /etc/grafana/dashboards/ai-platform-sla.json
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro
      - ./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards:ro
      - ./grafana/dashboards:/etc/grafana/dashboards:ro

volumes:
  prometheus_data:
  grafana_data:

prometheus.yml

yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: ai-platform-sla-exporter
    static_configs:
      - targets: ["host.docker.internal:8781"]
        labels:
          env: "207"
          platform: "workbuddy-ai"

Grafana 数据源

yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    uid: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

Grafana 大盘加载

yaml
apiVersion: 1

providers:
  - name: AI Platform SLA
    orgId: 1
    folder: Workbuddy
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/dashboards

大盘 JSON 完整文件:patches_cursor/作业8_SLA监控大盘/grafana/dashboards/ai-platform-sla.json(uid ai-platform-sla)。只维护这一份,避免文档里再贴一份导致漂移。

作业8_01_部署监控栈.sh

bash
#!/usr/bin/env bash
# 作业 8 · 在 207 上部署 Prometheus + Grafana 监控栈
set -euo pipefail

PATCH="/opt/workbuddy/patches_cursor/作业8_SLA监控大盘"
COMPOSE="${PATCH}/docker-compose.monitoring.yml"

if ! command -v docker >/dev/null 2>&1; then
  echo "[FAIL] docker 未安装"
  exit 1
fi

cd "$PATCH"
if docker compose version >/dev/null 2>&1; then
  DC="docker compose"
elif command -v docker-compose >/dev/null 2>&1; then
  DC="docker-compose"
else
  echo "[FAIL] 需要 docker compose 或 docker-compose"
  exit 1
fi

$DC -f "$COMPOSE" pull
# 跳过 pull 已缓存镜像时仍 recreate
$DC -f "$COMPOSE" up -d --force-recreate
sleep 10

echo "=== containers ==="
docker ps --filter name=hw8- --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

echo "=== prometheus health ==="
curl -sf "http://127.0.0.1:9092/-/healthy" && echo

echo "=== grafana health ==="
curl -sf "http://127.0.0.1:3001/api/health" && echo

echo "[OK] Grafana http://192.168.110.207:3001  admin/admin123"
echo "[OK] Prometheus http://192.168.110.207:9092"

作业8_启动_exporter.sh

bash
#!/usr/bin/env bash
# 作业 8 · 在 207 上启动 SLA Prometheus Exporter
set -euo pipefail

DIR="/opt/workbuddy/scripts_cursor/作业8_SLA监控大盘"
PORT="${SLA_EXPORTER_PORT:-8781}"
PIDFILE="/tmp/hw8_sla_exporter.pid"
LOG="/tmp/hw8_sla_exporter.log"

if [[ -f "$PIDFILE" ]] && kill -0 "$(cat "$PIDFILE")" 2>/dev/null; then
  if [[ "${FORCE_RESTART:-0}" == "1" ]]; then
    echo "restarting sla_exporter pid=$(cat "$PIDFILE")"
    kill "$(cat "$PIDFILE")" 2>/dev/null || true
    sleep 1
    rm -f "$PIDFILE"
  else
    echo "sla_exporter already running pid=$(cat "$PIDFILE")"
    curl -sf "http://127.0.0.1:${PORT}/health" && echo
    exit 0
  fi
fi

mkdir -p /opt/workbuddy/data/homework8
cd "$DIR"
nohup python3 作业8_sla_exporter.py --port "$PORT" --interval 60 >>"$LOG" 2>&1 &
echo $! >"$PIDFILE"
sleep 2
curl -sf "http://127.0.0.1:${PORT}/health" && echo
curl -sf "http://127.0.0.1:${PORT}/metrics" | head -n 8
echo "started on :${PORT}, log=$LOG"

作业8_01_部署监控栈.py

python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""作业 8 · 步骤 1:部署 Prometheus + Grafana。"""
from __future__ import annotations

import sys
from pathlib import Path

import _bootstrap  # noqa: F401
import _config as C
from dify_api import Dify

LOCAL_SH = (
    Path(__file__).resolve().parents[2]
    / "scripts_bash_cursor"
    / "作业8_SLA监控大盘"
    / "作业8_01_部署监控栈.sh"
)


def main() -> int:
    d = Dify(C.DIFY_EMAIL, C.DIFY_PASSWORD)
    try:
        d.login()
        d.sftp.put(str(LOCAL_SH), "/tmp/作业8_01_部署监控栈.sh")
        out, err = d.sh("bash /tmp/作业8_01_部署监控栈.sh")
        print(out or err)
        out2, _ = d.sh("curl -sf '%s/-/healthy'" % C.HOMEWORK8_PROMETHEUS_URL)
        if "Healthy" not in (out2 or ""):
            print("[FAIL] Prometheus 未就绪")
            return 1
        out3, _ = d.sh("curl -sf '%s/api/health'" % C.HOMEWORK8_GRAFANA_URL)
        if "ok" not in (out3 or "").lower():
            print("[FAIL] Grafana 未就绪")
            return 1
        print("[OK] 监控栈 %s / %s" % (C.HOMEWORK8_PROMETHEUS_URL, C.HOMEWORK8_GRAFANA_URL))
        return 0
    finally:
        d.close()


if __name__ == "__main__":
    sys.exit(main())

作业8_02_启动_exporter.py

python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""作业 8 · 步骤 2:启动 SLA Exporter(:8781)。"""
from __future__ import annotations

import sys
from pathlib import Path

import _bootstrap  # noqa: F401
import _config as C
from dify_api import Dify

LOCAL_SH = (
    Path(__file__).resolve().parents[2]
    / "scripts_bash_cursor"
    / "作业8_SLA监控大盘"
    / "作业8_启动_exporter.sh"
)


def main() -> int:
    d = Dify(C.DIFY_EMAIL, C.DIFY_PASSWORD)
    try:
        d.login()
        d.sftp.put(str(LOCAL_SH), "/tmp/作业8_启动_exporter.sh")
        out, err = d.sh("FORCE_RESTART=1 bash /tmp/作业8_启动_exporter.sh")
        print(out or err)
        out2, _ = d.sh("curl -sf '%s/metrics' | grep ai_platform_availability_ratio" % C.SLA_EXPORTER_BASE)
        if "ai_platform_availability_ratio" not in (out2 or ""):
            print("[FAIL] Exporter 未输出指标")
            return 1
        print("[OK] SLA Exporter %s/metrics" % C.SLA_EXPORTER_BASE)
        return 0
    finally:
        d.close()


if __name__ == "__main__":
    sys.exit(main())