feat(monitoring): add automatic restart of unreachable customer containers

Some customers only regain access to their NetBird web UI after a manual
container restart, and until now that required someone to notice and do it
by hand. The scheduler now optionally probes each customer's actual
dashboard URL over HTTP(S) every poll interval (not just Docker container
state, since a container can be "running" while the web UI behind it still
doesn't respond) and restarts that customer's containers if it doesn't
answer, gated by a per-deployment cooldown to avoid restart loops on a
persistently broken customer.

New system setting "Automatic Restart of Unreachable Containers" (default
off) with a configurable cooldown. Every automatic restart is logged in the
affected customer's own log, same as a manual restart.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

app/services/netbird_service.py

@@ -26,6 +26,7 @@ import urllib.error
 from datetime import datetime
 from typing import Any
 
+import httpx
 from jinja2 import Environment, FileSystemLoader
 from sqlalchemy.orm import Session
 
@@ -529,6 +530,25 @@ async def restart_customer(db: Session, customer_id: int) -> dict[str, Any]:
     return {"success": ok}
 
 
+async def check_dashboard_reachable(setup_url: str, timeout: float = 10) -> bool:
+    """Check whether a customer's NetBird dashboard actually responds over HTTP(S).
+
+    A container can be "running" in Docker while the web UI behind it is
+    still unreachable (e.g. the management service is up but not accepting
+    connections yet), so this checks the real endpoint rather than the
+    container state. Any HTTP response (including 4xx/5xx from the app
+    itself) counts as reachable — only connection failures/timeouts don't.
+    """
+    if not setup_url:
+        return True  # nothing to check (e.g. deployment not fully set up yet)
+    try:
+        async with httpx.AsyncClient(timeout=timeout) as client:
+            await client.get(setup_url)
+        return True
+    except httpx.HTTPError:
+        return False
+
+
 def get_customer_health(db: Session, customer_id: int) -> dict[str, Any]:
     """Check health of a customer's deployment."""
     deployment = db.query(Deployment).filter(Deployment.customer_id == customer_id).first()