集成环境部署:Prometheus + Grafana + SLA Exporter
本文定位:207 上 SLA 监控集成环境 的部署与配置教程(独立类目)。作业 8 讲「大盘长什么样、指标怎么算」;本文讲「三个服务怎么装、端口为什么这样分、配置文件写什么」。
环境:
192.168.110.207- Prometheus :9092
- Grafana :3001(账号
admin/admin123)- SLA Exporter :8781
Langfuse 仍占用 :3000,MinIO API :9090、MinIO Console 127.0.0.1:9091,不要抢。
HTML:
docs/html_cursor/集成环境部署_Prometheus_Grafana_SLA_cursor.html
大盘成品(作业 8):http://192.168.110.207:3001/d/ai-platform-sla
作业正文:作业8:SLA 监控大盘
目录
207 端口与角色
本机已经有一批服务。监控栈必须错开端口,否则 Docker 绑定失败(实战踩过::9091 已被 MinIO Console 占用)。
| 服务 | 地址 | 谁占用 | 本教程 |
|---|---|---|---|
| Dify | http://192.168.110.207(80) | 已有 | 只探测,不改 |
| Langfuse | :3000 | 已有 | 只探测,不改 |
| MinIO API | :9090 | 已有 | 不是 Prometheus |
| MinIO Console | 127.0.0.1:9091 | 已有 | 不要再绑 9091 |
| Xinference | :9997 | 已有 | 只探测 |
| 作业5 健康 API | :8779 | 已有 | 只探测 |
| 作业7 自愈 API | :8780 | 已有 | 不改 |
| SLA Exporter | :8781 | 本环境 | Python 进程,暴露 /metrics |
| Grafana | :3001 | 本环境 | 容器 hw8-grafana |
| Prometheus | :9092 | 本环境 | 容器 hw8-prometheus(容器内仍是 9090) |
_config.py 对应常量:
| 常量 | 值 |
|---|---|
SLA_EXPORTER_PORT / SLA_EXPORTER_BASE | 8781 / http://192.168.110.207:8781 |
HOMEWORK8_PROMETHEUS_PORT / _URL | 9092 / http://192.168.110.207:9092 |
HOMEWORK8_GRAFANA_PORT / _URL | 3001 / http://192.168.110.207:3001 |
HOMEWORK8_GRAFANA_USER / _PASS | admin / admin123 |
HOMEWORK8_DASHBOARD_UID | ai-platform-sla |
HOMEWORK8_PATCH_DIR | /opt/workbuddy/patches_cursor/作业8_SLA监控大盘 |
HOMEWORK8_DATA_DIR | /opt/workbuddy/data/homework8 |
Grafana 直达大盘:http://192.168.110.207:3001/d/ai-platform-sla
架构
207 宿主机
├─ python3 作业8_sla_exporter.py 0.0.0.0:8781
│ /health /metrics
│ 每 60s 健康检查 + Langfuse P99 + 演示项
│
├─ Docker hw8-prometheus 0.0.0.0:9092 → 容器:9090
│ scrape host.docker.internal:8781 每 15s
│
└─ Docker hw8-grafana 0.0.0.0:3001 → 容器:3000
数据源 http://prometheus:9090(compose 内网名)
预置大盘 uid=ai-platform-slaPrometheus 在容器里访问宿主机上的 Exporter,靠 extra_hosts: host.docker.internal:host-gateway。 Grafana 访问 Prometheus 用 compose 服务名 prometheus:9090,不要写 192.168.110.207:9092(容器内走 overlay 网更快、也更稳)。
目录与配置文件
全部挂在作业 8 子目录(程序与文档同步约定:按文档名分子目录)。
patches_cursor/作业8_SLA监控大盘/
├── docker-compose.monitoring.yml
├── prometheus/prometheus.yml
└── grafana/
├── dashboards/ai-platform-sla.json
└── provisioning/
├── datasources/prometheus.yml
└── dashboards/dashboards.yml
scripts_bash_cursor/作业8_SLA监控大盘/
├── 作业8_01_部署监控栈.sh # docker compose up
└── 作业8_启动_exporter.sh # nohup python3 :8781
scripts_cursor/作业8_SLA监控大盘/
├── 作业8_01_部署监控栈.py # Windows SSH 调上面的 sh
├── 作业8_02_启动_exporter.py
├── 作业8_sla_exporter.py
└── 作业8_sla_metrics.py一键部署
Windows 项目根(先同步再部署):
python scripts_cursor/00_部署_workbuddy到207.py
python scripts_cursor/作业8_SLA监控大盘/作业8_01_部署监控栈.py
python scripts_cursor/作业8_SLA监控大盘/作业8_02_启动_exporter.py| 脚本 | 干什么 | 成功标志 |
|---|---|---|
作业8_01_部署监控栈.py | SSH 207,docker compose up -d --force-recreate 起 hw8-prometheus / hw8-grafana | [OK] 监控栈 http://192.168.110.207:9092 / http://192.168.110.207:3001 |
作业8_02_启动_exporter.py | FORCE_RESTART=1 拉起 :8781 | [OK] SLA Exporter http://192.168.110.207:8781/metrics |
前置:207 已装 Docker,能 docker compose(或 docker-compose);/opt/workbuddy 已同步。
手工部署(207 本机)
4.1 起 Prometheus + Grafana
cd /opt/workbuddy/patches_cursor/作业8_SLA监控大盘
docker compose -f docker-compose.monitoring.yml up -d --force-recreate
curl -sf http://127.0.0.1:9092/-/healthy
curl -sf http://127.0.0.1:3001/api/health期望:Prometheus Server is Healthy. 以及 Grafana JSON 里 "database": "ok"。
4.2 起 SLA Exporter
FORCE_RESTART=1 bash /opt/workbuddy/scripts_bash_cursor/作业8_SLA监控大盘/作业8_启动_exporter.sh进程、日志:
| 项 | 路径 |
|---|---|
| PID | /tmp/hw8_sla_exporter.pid |
| 日志 | /tmp/hw8_sla_exporter.log |
| 状态文件 | /opt/workbuddy/data/homework8/sla_state.json |
改了 Python 必须 FORCE_RESTART=1,否则还是旧进程。
验收
curl -sf http://192.168.110.207:8781/health
curl -sf http://192.168.110.207:8781/metrics | grep -E 'ai_platform_component_up|name='
curl -sf http://192.168.110.207:9092/api/v1/targets
curl -sf http://192.168.110.207:3001/api/healthPrometheus Targets 里 job ai-platform-sla-exporter 应为 UP。 浏览器:http://192.168.110.207:3001/d/ai-platform-sla(admin / admin123)。 组件健康一行必须能看出是 Dify / Langfuse / vLLM / Xinference / 健康 API,不能只显示 UP。
Windows 再跑:
python scripts_cursor/作业8_SLA监控大盘/作业8_04_验收大盘.py成功:[PASS] 作业8 SLA 大盘验收。
日常运维
| 动作 | 命令 |
|---|---|
| 看容器 | docker ps --filter name=hw8- |
| 重启 Grafana(改过大盘 JSON 后) | docker restart hw8-grafana |
| 重建监控栈 | 重跑 作业8_01_部署监控栈.py |
| 重启 Exporter | FORCE_RESTART=1 bash .../作业8_启动_exporter.sh |
| 停监控栈 | cd /opt/workbuddy/patches_cursor/作业8_SLA监控大盘 && docker compose -f docker-compose.monitoring.yml down |
| 停 Exporter | kill $(cat /tmp/hw8_sla_exporter.pid) |
典型失败
| 现象 | 原因 | 处理 |
|---|---|---|
listen tcp 0.0.0.0:9091: address already in use | MinIO Console 占了 9091 | 本环境 Prometheus 必须用 9092,不要改回 9091 |
Grafana 容器 read-only file system 挂载失败 | 把 dashboard JSON 挂到另一个 :ro 目录的子路径 | 大盘挂在 /etc/grafana/dashboards,provisioning 只挂 yaml |
| Prometheus Target DOWN | Exporter 没起,或容器访问不到宿主机 8781 | 先 curl 127.0.0.1:8781/metrics;确认 compose 有 host.docker.internal:host-gateway |
| 大盘还是旧布局 | Grafana 没加载新 JSON | 同步后再 docker restart hw8-grafana,浏览器 Ctrl+F5 |
| Exporter 指标不变 | 旧 python 进程没杀 | FORCE_RESTART=1 重拉 |
完整配置
docker-compose.monitoring.yml
services:
prometheus:
image: prom/prometheus:v2.54.1
container_name: hw8-prometheus
restart: unless-stopped
extra_hosts:
- "host.docker.internal:host-gateway"
ports:
- "9092:9090"
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
command:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.retention.time=15d"
grafana:
image: grafana/grafana:11.2.0
container_name: hw8-grafana
restart: unless-stopped
depends_on:
- prometheus
ports:
- "3001:3000"
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin123
GF_USERS_ALLOW_SIGN_UP: "false"
GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /etc/grafana/dashboards/ai-platform-sla.json
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources:ro
- ./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards:ro
- ./grafana/dashboards:/etc/grafana/dashboards:ro
volumes:
prometheus_data:
grafana_data:prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: ai-platform-sla-exporter
static_configs:
- targets: ["host.docker.internal:8781"]
labels:
env: "207"
platform: "workbuddy-ai"Grafana 数据源
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
editable: falseGrafana 大盘加载
apiVersion: 1
providers:
- name: AI Platform SLA
orgId: 1
folder: Workbuddy
type: file
disableDeletion: false
updateIntervalSeconds: 30
options:
path: /etc/grafana/dashboards大盘 JSON 完整文件:patches_cursor/作业8_SLA监控大盘/grafana/dashboards/ai-platform-sla.json(uid ai-platform-sla)。只维护这一份,避免文档里再贴一份导致漂移。
作业8_01_部署监控栈.sh
#!/usr/bin/env bash
# 作业 8 · 在 207 上部署 Prometheus + Grafana 监控栈
set -euo pipefail
PATCH="/opt/workbuddy/patches_cursor/作业8_SLA监控大盘"
COMPOSE="${PATCH}/docker-compose.monitoring.yml"
if ! command -v docker >/dev/null 2>&1; then
echo "[FAIL] docker 未安装"
exit 1
fi
cd "$PATCH"
if docker compose version >/dev/null 2>&1; then
DC="docker compose"
elif command -v docker-compose >/dev/null 2>&1; then
DC="docker-compose"
else
echo "[FAIL] 需要 docker compose 或 docker-compose"
exit 1
fi
$DC -f "$COMPOSE" pull
# 跳过 pull 已缓存镜像时仍 recreate
$DC -f "$COMPOSE" up -d --force-recreate
sleep 10
echo "=== containers ==="
docker ps --filter name=hw8- --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
echo "=== prometheus health ==="
curl -sf "http://127.0.0.1:9092/-/healthy" && echo
echo "=== grafana health ==="
curl -sf "http://127.0.0.1:3001/api/health" && echo
echo "[OK] Grafana http://192.168.110.207:3001 admin/admin123"
echo "[OK] Prometheus http://192.168.110.207:9092"作业8_启动_exporter.sh
#!/usr/bin/env bash
# 作业 8 · 在 207 上启动 SLA Prometheus Exporter
set -euo pipefail
DIR="/opt/workbuddy/scripts_cursor/作业8_SLA监控大盘"
PORT="${SLA_EXPORTER_PORT:-8781}"
PIDFILE="/tmp/hw8_sla_exporter.pid"
LOG="/tmp/hw8_sla_exporter.log"
if [[ -f "$PIDFILE" ]] && kill -0 "$(cat "$PIDFILE")" 2>/dev/null; then
if [[ "${FORCE_RESTART:-0}" == "1" ]]; then
echo "restarting sla_exporter pid=$(cat "$PIDFILE")"
kill "$(cat "$PIDFILE")" 2>/dev/null || true
sleep 1
rm -f "$PIDFILE"
else
echo "sla_exporter already running pid=$(cat "$PIDFILE")"
curl -sf "http://127.0.0.1:${PORT}/health" && echo
exit 0
fi
fi
mkdir -p /opt/workbuddy/data/homework8
cd "$DIR"
nohup python3 作业8_sla_exporter.py --port "$PORT" --interval 60 >>"$LOG" 2>&1 &
echo $! >"$PIDFILE"
sleep 2
curl -sf "http://127.0.0.1:${PORT}/health" && echo
curl -sf "http://127.0.0.1:${PORT}/metrics" | head -n 8
echo "started on :${PORT}, log=$LOG"作业8_01_部署监控栈.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""作业 8 · 步骤 1:部署 Prometheus + Grafana。"""
from __future__ import annotations
import sys
from pathlib import Path
import _bootstrap # noqa: F401
import _config as C
from dify_api import Dify
LOCAL_SH = (
Path(__file__).resolve().parents[2]
/ "scripts_bash_cursor"
/ "作业8_SLA监控大盘"
/ "作业8_01_部署监控栈.sh"
)
def main() -> int:
d = Dify(C.DIFY_EMAIL, C.DIFY_PASSWORD)
try:
d.login()
d.sftp.put(str(LOCAL_SH), "/tmp/作业8_01_部署监控栈.sh")
out, err = d.sh("bash /tmp/作业8_01_部署监控栈.sh")
print(out or err)
out2, _ = d.sh("curl -sf '%s/-/healthy'" % C.HOMEWORK8_PROMETHEUS_URL)
if "Healthy" not in (out2 or ""):
print("[FAIL] Prometheus 未就绪")
return 1
out3, _ = d.sh("curl -sf '%s/api/health'" % C.HOMEWORK8_GRAFANA_URL)
if "ok" not in (out3 or "").lower():
print("[FAIL] Grafana 未就绪")
return 1
print("[OK] 监控栈 %s / %s" % (C.HOMEWORK8_PROMETHEUS_URL, C.HOMEWORK8_GRAFANA_URL))
return 0
finally:
d.close()
if __name__ == "__main__":
sys.exit(main())作业8_02_启动_exporter.py
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""作业 8 · 步骤 2:启动 SLA Exporter(:8781)。"""
from __future__ import annotations
import sys
from pathlib import Path
import _bootstrap # noqa: F401
import _config as C
from dify_api import Dify
LOCAL_SH = (
Path(__file__).resolve().parents[2]
/ "scripts_bash_cursor"
/ "作业8_SLA监控大盘"
/ "作业8_启动_exporter.sh"
)
def main() -> int:
d = Dify(C.DIFY_EMAIL, C.DIFY_PASSWORD)
try:
d.login()
d.sftp.put(str(LOCAL_SH), "/tmp/作业8_启动_exporter.sh")
out, err = d.sh("FORCE_RESTART=1 bash /tmp/作业8_启动_exporter.sh")
print(out or err)
out2, _ = d.sh("curl -sf '%s/metrics' | grep ai_platform_availability_ratio" % C.SLA_EXPORTER_BASE)
if "ai_platform_availability_ratio" not in (out2 or ""):
print("[FAIL] Exporter 未输出指标")
return 1
print("[OK] SLA Exporter %s/metrics" % C.SLA_EXPORTER_BASE)
return 0
finally:
d.close()
if __name__ == "__main__":
sys.exit(main())