Prometheus Port Conflict Issue

Problem

thread 'main' (21677) panicked at src/systems/telemetry.rs:57:10:
failed to install Prometheus recorder: FailedToCreateHTTPListener("Address already in use (os error 98)")

Root Cause

1. The game process terminates (crash, Ctrl+C, or normal exit)
2. The OS keeps the TCP socket in TIME_WAIT state for 30-120 seconds
3. Restarting the game immediately tries to bind to port 9000 again
4. The bind fails because the port is still in use

let builder = PrometheusBuilder::new();
builder
    .with_http_listener(([0, 0, 0, 0], 9000))
    .install()
    .expect("failed to install Prometheus recorder"); // ← Panics here

Solutions

Solution 1: Graceful Error Handling (Recommended for Development)

let builder = PrometheusBuilder::new();
builder
    .with_http_listener(([0, 0, 0, 0], 9000))
    .install()
    .expect("failed to install Prometheus recorder");

let builder = PrometheusBuilder::new();
match builder.with_http_listener(([0, 0, 0, 0], 9000)).install() {
    Ok(_) => {
        info!("Telemetry initialized. Metrics at http://localhost:9000/metrics");
    }
    Err(e) => {
        warn!("Failed to install Prometheus recorder: {}. Metrics will be unavailable.", e);
        warn!("This usually means port 9000 is already in use. Wait 30s or kill the process using the port.");
    }
}

• Game doesn't crash on restart
• Clear warning message to developers
• Metrics are optional, not critical for gameplay

• Metrics unavailable until port is freed
• Silent failure if you don't check logs

Solution 2: Dynamic Port Allocation

let builder = PrometheusBuilder::new();
match builder.with_http_listener(([0, 0, 0, 0], 0)).install() { // Port 0 = dynamic
    Ok(_) => {
        // Note: You won't know which port was assigned without additional code
        info!("Telemetry initialized. Metrics available on a dynamic port.");
    }
    Err(e) => {
        warn!("Failed to install Prometheus recorder: {}", e);
    }
}

• Never conflicts
• Works for multiple instances

• Port number is unknown (need to query the listener to find it)
• Harder to configure Prometheus scraper

Solution 3: SO_REUSEADDR Socket Option

1. Fork metrics-exporter-prometheus
2. Add SO_REUSEADDR to the socket before binding
3. Use your forked version

• Immediate port reuse
• Consistent port number

• Requires maintaining a fork
• More complex

Solution 4: Kill Existing Process on Port 9000

# Find process using port 9000
lsof -ti:9000

# Kill it
kill -9 $(lsof -ti:9000)

# Then run the game
cargo run --bin legends_client

#!/bin/bash
# kill_and_run.sh
lsof -ti:9000 | xargs -r kill -9
cargo run --bin legends_client

• Ensures port is always available
• No code changes needed

• Might kill unrelated processes
• Platform-specific (Linux/macOS only)

Solution 5: Wait for Port to Be Released

let builder = PrometheusBuilder::new();
let mut retries = 0;
let max_retries = 3;

loop {
    match builder.with_http_listener(([0, 0, 0, 0], 9000)).install() {
        Ok(_) => {
            info!("Telemetry initialized. Metrics at http://localhost:9000/metrics");
            break;
        }
        Err(e) if retries < max_retries => {
            retries += 1;
            warn!("Failed to bind port 9000 (attempt {}/{}): {}. Retrying in 2s...", retries, max_retries, e);
            std::thread::sleep(std::time::Duration::from_secs(2));
        }
        Err(e) => {
            warn!("Failed to install Prometheus recorder after {} attempts: {}", max_retries, e);
            break;
        }
    }
}

• Automatic recovery
• No manual intervention

• Delays startup
• Still might fail if port is held by another service

Recommended Approach

• Fast iteration
• Non-blocking
• Clear error messages

• Deploy with a sidecar that queries the dynamic port
• Register with Prometheus via service discovery (Kubernetes, Consul, etc.)

• Convenient for rapid testing
• No code changes

Implemented Solution

let builder = PrometheusBuilder::new();
builder
    .with_http_listener(([0, 0, 0, 0], 9000))
    .install()
    .expect("failed to install Prometheus recorder");
    
info!("Telemetry initialized. Metrics at http://localhost:9000/metrics");

let builder = PrometheusBuilder::new();
builder
    .with_http_listener(([0, 0, 0, 0], 9000))
    .install()
    .unwrap_or_else(|e| {
        warn!("Failed to start Prometheus exporter (port 9000 likely in use): {}", e);
    });
    
info!("Telemetry initialized. Metrics attempted at http://localhost:9000/metrics");

1. Replaced .expect() with .unwrap_or_else() to handle errors gracefully
2. Log a warning instead of panicking when port binding fails
3. Updated success message to say "attempted" instead of "at" to reflect that metrics may not be available

Verification

# First run
cargo run --bin legends_client
# Output: Telemetry initialized. Metrics attempted at http://localhost:9000/metrics

# Ctrl+C to stop

# Immediate second run
cargo run --bin legends_client
# Output (if port still in use):
# WARN Failed to start Prometheus exporter (port 9000 likely in use): FailedToCreateHTTPListener("Address already in use (os error 98)")
# INFO Telemetry initialized. Metrics attempted at http://localhost:9000/metrics
# Game continues to run normally ✅

Related Issues

• Tracing panic: See tracing_panic_fix.md
• Duplicate plugin panic: See duplicate_plugin_panic.md

Platform-Specific Notes

Linux

• Default TIME_WAIT duration: 60 seconds
• Check port usage: lsof -i :9000 or ss -tuln | grep 9000

macOS

• Default TIME_WAIT duration: 15 seconds
• Check port usage: lsof -i :9000

Windows

• Default TIME_WAIT duration: 120 seconds
• Check port usage: netstat -ano | findstr :9000