observability

Guide for observability

Observability Stack

Local development observability using Prometheus and Grafana.

For instructions on generating load to see these metrics in action, see Load Testing Guide.

Location

loh-ops-tools/observability/

Quick Start

cd loh-ops-tools/observability
docker compose up -d

Service	URL	Credentials
Prometheus	http://localhost:9090	-
Grafana	http://localhost:6006	admin / admin

Components

Prometheus: Metrics collection (scrapes /metrics every 5s)
Grafana: Dashboard visualization (pre-provisioned)
Node Exporter: Host metrics (CPU, Memory, Disk, Network)

Game Server Integration

The api-game crate exposes metrics at GET /metrics:

Metric	Type	Description
`loh_connected_players`	Gauge	Current WebSocket connections
`loh_game_tick_duration`	Histogram	Game loop tick latency (ms)
`loh_messages_processed_total`	Counter	Total messages handled
`loh_messages_by_type`	Counter	Messages by type label
`loh_ws_connections_total`	Counter	Total WS connections
`loh_ws_disconnections_total`	Counter	Total WS disconnections
`loh_db_query_duration`	Histogram	Database query latency

Usage in Code

use api_game::metrics;

// In game loop:
metrics::set_connected_players(count);
metrics::record_tick_duration(elapsed_ms);
metrics::increment_messages_processed();

SRE Dashboard Panels

The pre-built loh-sre-dashboard.json includes:

CPU Usage (%)
Memory Usage
Connected Players (live gauge)
Game Tick Duration (p50/p99)
Network I/O (RX/TX)
Messages per Second

Files

observability/
├── docker-compose.yml
├── prometheus/
│   └── prometheus.yml
├── grafana/
│   ├── dashboards/
│   │   └── loh-sre-dashboard.json
│   └── provisioning/
│       ├── dashboards/dashboards.yml
│       └── datasources/datasources.yml
└── README.md

Cleanup

docker compose down -v