Compare commits

...
17 Commits
Author SHA1 Message Date
twothatITandClaude Sonnet 5 419598220a chore(release): bump version to 1.6.2
Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-10-20 08:47:40 +02:00
twothatITandClaude Sonnet 5 565dc5f710 fix(auth): replace unmaintained python-jose with PyJWT
Dependabot alert #25 (python-jose algorithm confusion guard bypassed by
DER-encoded public keys, CVE-2026-85394, Critical) has no patched version
of python-jose available at all - it's affected through the latest
release 3.5.0. python-jose was only used in app/dependencies.py for plain
HS256 JWT encode/decode with our own symmetric SECRET_KEY (algorithms=[...]
was already explicitly restricted on every decode call, so the specific
RS256/HS256 confusion path didn't apply here), but staying on an
unmaintained library that keeps generating unfixable critical alerts isn't
sustainable.

Swapped to PyJWT, which is actively maintained, already pulled in
transitively via msal, and a near drop-in replacement for this app's
encode()/decode() usage. Verified with a Docker build and in-container
token create/verify/reject round-trip.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-10-20 08:47:33 +02:00
twothatITandClaude Sonnet 5 4078295ea9 chore(release): bump version to 1.6.1
Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-10-20 08:42:12 +02:00
twothatITandClaude Sonnet 5 6b5cbc722f fix(deps): resolve all 25 open Dependabot alerts
Bumps every flagged dependency to its patched (or latest compatible)
version: python-jose 3.3.0->3.4.0 (2 critical algorithm-confusion alerts),
cryptography 41.0.7->50.0.0 (7 alerts incl. a Bleichenbacher timing oracle
and outdated bundled OpenSSL), python-multipart 0.0.6->0.0.31 (9 alerts,
incl. a high-severity arbitrary file write), Jinja2 3.1.2->3.1.6 (5 sandbox
breakout / HTML injection alerts), and pytest 7.4.3->9.0.3 (1 alert,
dev-only).

msal had to move 1.28.0->1.39.0 because it pinned cryptography<45;
httpx had to move 0.26.0->0.28.1 and pytest-asyncio 0.23.2->1.4.0 and
pytest-httpx 0.28.0->0.36.2 to keep the dependency set resolvable
together with the above. Verified with a full Docker build and a running
container (health check, settings load, DB migrations all green).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-10-20 08:42:03 +02:00
twothatITandClaude Sonnet 5 8aa446a9c2 chore(release): bump version to 1.6.0
Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-09-23 13:20:48 +02:00
twothatITandClaude Sonnet 5 189c49d505 feat(monitoring): add automatic restart of unreachable customer containers
Some customers only regain access to their NetBird web UI after a manual
container restart, and until now that required someone to notice and do it
by hand. The scheduler now optionally probes each customer's actual
dashboard URL over HTTP(S) every poll interval (not just Docker container
state, since a container can be "running" while the web UI behind it still
doesn't respond) and restarts that customer's containers if it doesn't
answer, gated by a per-deployment cooldown to avoid restart loops on a
persistently broken customer.

New system setting "Automatic Restart of Unreachable Containers" (default
off) with a configurable cooldown. Every automatic restart is logged in the
affected customer's own log, same as a manual restart.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-09-23 13:20:40 +02:00
twothatIT 75357b0633 fix(monitoring): fix untranslated "common.refresh" key on Update Log button
Used a translation key that doesn't exist (common.refresh) instead of the
existing monitoring.refresh, so the button rendered the raw key instead of
"Aktualisieren"/"Refresh".
2026-08-21 22:53:37 +02:00
twothatIT 556db5bfd4 chore(release): bump version to 1.5.0 2026-08-21 22:40:22 +02:00
twothatIT d4b6efeea1 feat(monitoring): add update-run log, fix scheduler skipping stale customer containers
The automatic scheduler only recreated customer containers when it had just
found a NEW image on Docker Hub. If an image was already pulled locally
(e.g. from an earlier manual pull, or auto-apply was off at the time) but
customer containers hadn't been recreated yet, the daily check reported
"already up to date" and silently never touched those customers again —
the recreate step needs to run independently of whether Hub had anything
new, based on each customer's actual container-vs-local-image state.

Also adds a persistent UpdateRunLog: every scheduled and manual pull/update
run now records per-image pull results and per-customer recreate results,
visible in Monitoring > Update Log, instead of only existing in container
logs.
2026-08-21 22:40:12 +02:00
twothatIT 276f4214a7 chore(release): bump version to 1.4.2 2026-08-19 11:22:08 +02:00
twothatIT a8ed87faa9 feat(customer): show NetBird API token renewal/expiry date
Displays when the per-customer token was last (re-)issued and its
computed expiry (365 days from renewal) on the Auto-Updates card, so it's
visible at a glance without checking the database.
2026-08-19 11:21:55 +02:00
twothatIT 1bc768151a feat(customer): allow changing an already-registered NetBird API token
The per-customer Auto-Updates card only showed the token input when no
token was registered yet, so there was no way to replace one once set
(e.g. rotating it manually). Adds a "Change token" link that reveals the
same input/save flow at any time.
2026-08-19 11:21:10 +02:00
twothatIT 1e05ddfe69 chore(release): bump version to 1.4.1 2026-08-19 11:17:38 +02:00
twothatIT e9afdb226c feat(netbird): auto-renew NetBird client-update API tokens before expiry
The tokens captured for central update control expire (NetBird enforces a
365-day max on Personal Access Tokens), and nothing was renewing them —
discovered that the 47 tokens created via the browser-automation bulk
onboarding were actually only 30-day tokens (left the UI's default
expiration field untouched instead of setting 365), so they would have
silently broken automatic-update control next month with no warning.

- Bumped all existing tokens to fresh 365-day ones via the API (using the
  still-valid old token as bearer — no re-login needed)
- Added netbird_api_token_renewed_at per deployment
- Scheduler now checks daily and renews any token older than 300 days
  automatically, so this never needs to be done by hand again
2026-08-19 11:15:49 +02:00
twothatIT 8a84e60a27 fix(validators): restore SystemConfigUpdate validators broken by misplaced classes
The two new NetbirdClientAutoUpdatePayload/NetbirdApiTokenPayload classes
were accidentally inserted between two of SystemConfigUpdate's own
@field_validator methods, which closed that class early — the ssl_mode,
base_domain, npm_api_url, and admin_email validators ended up attached to
NetbirdApiTokenPayload instead, whose fields don't exist. Pydantic raised
PydanticUserError on import, crash-looping the whole appliance container.
Moved the two new classes after all of SystemConfigUpdate's validators.
2026-08-19 09:35:12 +02:00
twothatIT a91a95825a chore(release): bump version to 1.4.0 2026-08-19 09:29:47 +02:00
twothatIT 6d333223a8 feat(netbird): central control of client Automatic Updates across all customers
Lets the MSP admin control NetBird's own "Settings > Clients > Automatic
Updates" feature (client/peer auto-update, v0.61.0+) for every customer from
one place, instead of logging into each customer's dashboard individually.

- New deployments automatically capture a Personal Access Token during the
  existing /api/setup bootstrap call (create_pat=true), requiring
  NB_SETUP_PAT_ENABLED=true on the management container (now set by default
  in the compose template). Token is encrypted at rest per customer.
- Existing customers (deployed before this existed) can have a token pasted
  in manually from their own dashboard — verified before being stored.
- Settings > Docker Images: master default (version + force-update toggle)
  plus "Apply to All Customers" which pushes it to everyone with a token.
- Customer detail page: shows the customer's live current setting (read
  from their NetBird API, not cached) with per-customer override or
  "sync from default".
- New app/services/netbird_client_update_service.py wraps the customer's
  NetBird Management API (GET/PUT /api/accounts) for this.
2026-08-19 09:24:13 +02:00
17 changed files with 1256 additions and 50 deletions
+10
View File
@@ -45,6 +45,7 @@ def init_db() -> None:
Deployment, Deployment,
DeploymentLog, DeploymentLog,
SystemConfig, SystemConfig,
UpdateRunLog,
User, User,
) )
@@ -127,6 +128,15 @@ def _run_migrations() -> None:
("system_config", "auto_update_check_time", "TEXT DEFAULT '03:00'"), ("system_config", "auto_update_check_time", "TEXT DEFAULT '03:00'"),
("system_config", "auto_update_apply_enabled", "BOOLEAN DEFAULT 0"), ("system_config", "auto_update_apply_enabled", "BOOLEAN DEFAULT 0"),
("system_config", "auto_update_last_run_at", "TEXT"), ("system_config", "auto_update_last_run_at", "TEXT"),
# NetBird client (peer) automatic-updates master default + per-customer PAT
("deployments", "netbird_api_token_encrypted", "TEXT"),
("deployments", "netbird_api_token_renewed_at", "TEXT"),
("system_config", "netbird_client_auto_update_version", "TEXT DEFAULT 'disabled'"),
("system_config", "netbird_client_auto_update_always", "BOOLEAN DEFAULT 0"),
# Automatic restart of unreachable customer containers
("system_config", "auto_restart_unreachable_enabled", "BOOLEAN DEFAULT 0"),
("system_config", "auto_restart_cooldown_minutes", "INTEGER DEFAULT 30"),
("deployments", "last_auto_restart_at", "TEXT"),
] ]
for table, column, col_type in migrations: for table, column, col_type in migrations:
if not _has_column(table, column): if not _has_column(table, column):
+4 -3
View File
@@ -3,9 +3,10 @@
from datetime import datetime, timedelta from datetime import datetime, timedelta
from typing import Optional from typing import Optional
import jwt
from fastapi import Depends, HTTPException, status from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from jose import JWTError, jwt from jwt import PyJWTError
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from app.database import get_db from app.database import get_db
@@ -56,7 +57,7 @@ def verify_mfa_token(token: str) -> str:
detail="Invalid MFA token.", detail="Invalid MFA token.",
) )
return username return username
except JWTError: except PyJWTError:
raise HTTPException( raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED, status_code=status.HTTP_401_UNAUTHORIZED,
detail="MFA token expired or invalid.", detail="MFA token expired or invalid.",
@@ -95,7 +96,7 @@ def get_current_user(
status_code=status.HTTP_401_UNAUTHORIZED, status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid token payload.", detail="Invalid token payload.",
) )
except JWTError: except PyJWTError:
raise HTTPException( raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED, status_code=status.HTTP_401_UNAUTHORIZED,
detail="Invalid or expired token.", detail="Invalid or expired token.",
+1 -1
View File
@@ -34,7 +34,7 @@ logger = logging.getLogger(__name__)
app = FastAPI( app = FastAPI(
title="NetBird MSP Appliance", title="NetBird MSP Appliance",
description="Multi-tenant NetBird management platform for MSPs", description="Multi-tenant NetBird management platform for MSPs",
version="1.3.0", version="1.6.2",
docs_url="/api/docs", docs_url="/api/docs",
redoc_url="/api/redoc", redoc_url="/api/redoc",
openapi_url="/api/openapi.json", openapi_url="/api/openapi.json",
+75
View File
@@ -1,5 +1,6 @@
"""SQLAlchemy ORM models for NetBird MSP Appliance.""" """SQLAlchemy ORM models for NetBird MSP Appliance."""
import json
from datetime import datetime from datetime import datetime
from typing import Optional from typing import Optional
@@ -88,11 +89,14 @@ class Deployment(Base):
setup_url: Mapped[Optional[str]] = mapped_column(Text, nullable=True) setup_url: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
netbird_admin_email: Mapped[Optional[str]] = mapped_column(Text, nullable=True) netbird_admin_email: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
netbird_admin_password: Mapped[Optional[str]] = mapped_column(Text, nullable=True) netbird_admin_password: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
netbird_api_token_encrypted: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
netbird_api_token_renewed_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
deployment_status: Mapped[str] = mapped_column( deployment_status: Mapped[str] = mapped_column(
String(20), default="pending", nullable=False String(20), default="pending", nullable=False
) )
deployed_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow) deployed_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
last_health_check: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) last_health_check: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
last_auto_restart_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
__table_args__ = ( __table_args__ = (
CheckConstraint( CheckConstraint(
@@ -116,11 +120,18 @@ class Deployment(Base):
"relay_secret": "***", # Never expose secrets "relay_secret": "***", # Never expose secrets
"setup_url": self.setup_url, "setup_url": self.setup_url,
"has_credentials": bool(self.netbird_admin_email and self.netbird_admin_password), "has_credentials": bool(self.netbird_admin_email and self.netbird_admin_password),
"has_netbird_api_token": bool(self.netbird_api_token_encrypted),
"netbird_api_token_renewed_at": (
self.netbird_api_token_renewed_at.isoformat() if self.netbird_api_token_renewed_at else None
),
"deployment_status": self.deployment_status, "deployment_status": self.deployment_status,
"deployed_at": self.deployed_at.isoformat() if self.deployed_at else None, "deployed_at": self.deployed_at.isoformat() if self.deployed_at else None,
"last_health_check": ( "last_health_check": (
self.last_health_check.isoformat() if self.last_health_check else None self.last_health_check.isoformat() if self.last_health_check else None
), ),
"last_auto_restart_at": (
self.last_auto_restart_at.isoformat() if self.last_auto_restart_at else None
),
} }
@@ -205,6 +216,16 @@ class SystemConfig(Base):
auto_update_apply_enabled: Mapped[bool] = mapped_column(Boolean, default=False) auto_update_apply_enabled: Mapped[bool] = mapped_column(Boolean, default=False)
auto_update_last_run_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True) auto_update_last_run_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
# Automatic restart of customer containers when found unreachable/not running
auto_restart_unreachable_enabled: Mapped[bool] = mapped_column(Boolean, default=False)
auto_restart_cooldown_minutes: Mapped[int] = mapped_column(Integer, default=30)
# Master default for the NetBird *client* (peer) automatic-updates feature
# (Settings > Clients > Automatic Updates inside each customer's own
# NetBird dashboard) — pushed to customers via the NetBird Management API.
netbird_client_auto_update_version: Mapped[str] = mapped_column(String(50), default="disabled")
netbird_client_auto_update_always: Mapped[bool] = mapped_column(Boolean, default=False)
created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow) created_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime, default=datetime.utcnow, onupdate=datetime.utcnow DateTime, default=datetime.utcnow, onupdate=datetime.utcnow
@@ -265,6 +286,10 @@ class SystemConfig(Base):
"auto_update_last_run_at": ( "auto_update_last_run_at": (
self.auto_update_last_run_at.isoformat() if self.auto_update_last_run_at else None self.auto_update_last_run_at.isoformat() if self.auto_update_last_run_at else None
), ),
"auto_restart_unreachable_enabled": bool(self.auto_restart_unreachable_enabled),
"auto_restart_cooldown_minutes": self.auto_restart_cooldown_minutes or 30,
"netbird_client_auto_update_version": self.netbird_client_auto_update_version or "disabled",
"netbird_client_auto_update_always": bool(self.netbird_client_auto_update_always),
"created_at": self.created_at.isoformat() if self.created_at else None, "created_at": self.created_at.isoformat() if self.created_at else None,
"updated_at": self.updated_at.isoformat() if self.updated_at else None, "updated_at": self.updated_at.isoformat() if self.updated_at else None,
} }
@@ -307,6 +332,56 @@ class DeploymentLog(Base):
} }
class UpdateRunLog(Base):
"""Audit trail for NetBird Docker image update runs (pull + container recreate).
Written for both scheduled (automatic) and manual (button-triggered)
runs, so the actual outcome — which image pulls failed, which customer
container recreates failed — is visible in the UI instead of only in
container logs.
"""
__tablename__ = "update_run_logs"
id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True)
run_type: Mapped[str] = mapped_column(String(20), nullable=False) # "scheduled" | "manual"
trigger: Mapped[Optional[str]] = mapped_column(String(255), nullable=True) # username, or "scheduler"
started_at: Mapped[datetime] = mapped_column(DateTime, default=datetime.utcnow)
finished_at: Mapped[Optional[datetime]] = mapped_column(DateTime, nullable=True)
status: Mapped[str] = mapped_column(
String(20), default="running"
) # running | no_update | success | partial | failed
any_update_available: Mapped[Optional[bool]] = mapped_column(Boolean, nullable=True)
pull_attempted: Mapped[bool] = mapped_column(Boolean, default=False)
pull_success: Mapped[Optional[bool]] = mapped_column(Boolean, nullable=True)
pull_details: Mapped[Optional[str]] = mapped_column(Text, nullable=True) # JSON: {image: {success, error}}
apply_attempted: Mapped[bool] = mapped_column(Boolean, default=False)
customers_updated: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
customers_total: Mapped[Optional[int]] = mapped_column(Integer, nullable=True)
customer_details: Mapped[Optional[str]] = mapped_column(Text, nullable=True) # JSON list
error: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
def to_dict(self) -> dict:
"""Serialize to dict, parsing the embedded JSON blobs."""
return {
"id": self.id,
"run_type": self.run_type,
"trigger": self.trigger,
"started_at": self.started_at.isoformat() if self.started_at else None,
"finished_at": self.finished_at.isoformat() if self.finished_at else None,
"status": self.status,
"any_update_available": self.any_update_available,
"pull_attempted": self.pull_attempted,
"pull_success": self.pull_success,
"pull_details": json.loads(self.pull_details) if self.pull_details else None,
"apply_attempted": self.apply_attempted,
"customers_updated": self.customers_updated,
"customers_total": self.customers_total,
"customer_details": json.loads(self.customer_details) if self.customer_details else None,
"error": self.error,
}
class User(Base): class User(Base):
"""Admin user model.""" """Admin user model."""
+124 -2
View File
@@ -1,6 +1,7 @@
"""Deployment management API — start, stop, restart, logs, health for customers.""" """Deployment management API — start, stop, restart, logs, health for customers."""
import logging import logging
from datetime import datetime
from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Query, status from fastapi import APIRouter, BackgroundTasks, Depends, HTTPException, Query, status
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -8,8 +9,9 @@ from sqlalchemy.orm import Session
from app.database import SessionLocal, get_db from app.database import SessionLocal, get_db
from app.dependencies import get_current_user from app.dependencies import get_current_user
from app.models import Customer, Deployment, SystemConfig, User from app.models import Customer, Deployment, SystemConfig, User
from app.services import docker_service, image_service, netbird_service from app.services import docker_service, image_service, netbird_client_update_service, netbird_service, update_log_service
from app.utils.security import decrypt_value from app.utils.security import decrypt_value, encrypt_value
from app.utils.validators import NetbirdApiTokenPayload, NetbirdClientAutoUpdatePayload
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
router = APIRouter() router = APIRouter()
@@ -244,9 +246,18 @@ async def update_customer_images(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="System not configured." status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="System not configured."
) )
started_at = datetime.utcnow()
instance_dir = f"{config.data_dir}/{customer.subdomain}" instance_dir = f"{config.data_dir}/{customer.subdomain}"
result = await image_service.update_customer_containers(instance_dir, deployment.container_prefix) result = await image_service.update_customer_containers(instance_dir, deployment.container_prefix)
update_log_service.record_run(
db, run_type="manual", trigger=current_user.username, started_at=started_at,
apply_attempted=True,
customer_results=[{
"customer_name": customer.name, "success": result["success"], "error": result.get("error"),
}],
)
if not result["success"]: if not result["success"]:
raise HTTPException( raise HTTPException(
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
@@ -260,6 +271,117 @@ async def update_customer_images(
return {"message": f"Containers updated for '{customer.name}'."} return {"message": f"Containers updated for '{customer.name}'."}
@router.get("/{customer_id}/netbird-updates")
async def get_customer_netbird_updates(
customer_id: int,
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
):
"""Fetch a customer's *live* NetBird client automatic-updates setting.
Reads directly from the customer's NetBird Management API — always
reflects reality, including changes made manually in their own dashboard.
"""
_require_customer(db, customer_id)
deployment = db.query(Deployment).filter(Deployment.customer_id == customer_id).first()
if not deployment:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="No deployment found for this customer.")
if not deployment.netbird_api_token_encrypted:
return {"has_token": False, "version": None, "always": None}
token = decrypt_value(deployment.netbird_api_token_encrypted)
result = await netbird_client_update_service.get_current_settings(deployment.container_prefix, token)
if not result["ok"]:
raise HTTPException(status_code=status.HTTP_502_BAD_GATEWAY, detail=result["error"])
settings = result["settings"]
return {
"has_token": True,
"token_renewed_at": (
deployment.netbird_api_token_renewed_at.isoformat()
if deployment.netbird_api_token_renewed_at else None
),
"version": settings.get("auto_update_version", "disabled"),
"always": bool(settings.get("auto_update_always", False)),
}
@router.put("/{customer_id}/netbird-updates")
async def set_customer_netbird_updates(
customer_id: int,
payload: NetbirdClientAutoUpdatePayload,
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
):
"""Push a client automatic-updates version/mode to a single customer.
Use this to override the master default for one customer specifically —
e.g. a customer on a legacy client that must not jump straight to latest.
"""
if current_user.role != "admin":
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin only.")
_require_customer(db, customer_id)
deployment = db.query(Deployment).filter(Deployment.customer_id == customer_id).first()
if not deployment:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="No deployment found for this customer.")
if not deployment.netbird_api_token_encrypted:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="No NetBird API token registered for this customer. Paste one via PUT .../netbird-api-token first.",
)
token = decrypt_value(deployment.netbird_api_token_encrypted)
result = await netbird_client_update_service.push_auto_update_settings(
deployment.container_prefix, token, payload.version, payload.always
)
if not result["ok"]:
raise HTTPException(status_code=status.HTTP_502_BAD_GATEWAY, detail=result["error"])
logger.info(
"NetBird client auto-update set for customer %d (%s): version=%s always=%s by %s",
customer_id, deployment.container_prefix, payload.version, payload.always, current_user.username,
)
return {"ok": True}
@router.put("/{customer_id}/netbird-api-token")
async def set_customer_netbird_api_token(
customer_id: int,
payload: NetbirdApiTokenPayload,
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
):
"""Manually register a NetBird Personal Access Token for a customer.
Needed for customers deployed before automatic PAT capture — create a
PAT once in that customer's dashboard (Settings > Service Users /
Personal Access Tokens) and paste it here. New deployments capture one
automatically during setup.
"""
if current_user.role != "admin":
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin only.")
_require_customer(db, customer_id)
deployment = db.query(Deployment).filter(Deployment.customer_id == customer_id).first()
if not deployment:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="No deployment found for this customer.")
# Validate the token actually works before storing it.
result = await netbird_client_update_service.get_current_settings(deployment.container_prefix, payload.token)
if not result["ok"]:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=f"Token could not be verified against this customer's NetBird instance: {result['error']}",
)
deployment.netbird_api_token_encrypted = encrypt_value(payload.token)
deployment.netbird_api_token_renewed_at = datetime.utcnow()
db.commit()
logger.info("NetBird API token registered for customer %d by %s.", customer_id, current_user.username)
return {"ok": True}
def _require_customer(db: Session, customer_id: int) -> Customer: def _require_customer(db: Session, customer_id: int) -> Customer:
"""Helper to fetch a customer or raise 404. """Helper to fetch a customer or raise 404.
+92 -5
View File
@@ -4,6 +4,7 @@ import asyncio
import logging import logging
import platform import platform
import time import time
from datetime import datetime
from typing import Any from typing import Any
import psutil import psutil
@@ -12,8 +13,10 @@ from sqlalchemy.orm import Session
from app.database import SessionLocal, get_db from app.database import SessionLocal, get_db
from app.dependencies import get_current_user from app.dependencies import get_current_user
from app.models import Customer, Deployment, SystemConfig, User from app.models import Customer, Deployment, SystemConfig, UpdateRunLog, User
from app.services import docker_service, image_service from app.services import docker_service, image_service, netbird_client_update_service, update_log_service
from app.utils.security import decrypt_value
from app.utils.validators import NetbirdClientAutoUpdatePayload
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
router = APIRouter() router = APIRouter()
@@ -192,14 +195,25 @@ async def pull_all_netbird_images(
config.netbird_dashboard_image, config.netbird_dashboard_image,
] ]
username = current_user.username
async def _pull_bg() -> None: async def _pull_bg() -> None:
started_at = datetime.utcnow()
bg_db = SessionLocal() bg_db = SessionLocal()
try: try:
cfg = bg_db.query(SystemConfig).filter(SystemConfig.id == 1).first() cfg = bg_db.query(SystemConfig).filter(SystemConfig.id == 1).first()
if cfg: if not cfg:
await image_service.pull_all_images(cfg) return
except Exception: pull_result = await image_service.pull_all_images(cfg)
update_log_service.record_run(
bg_db, run_type="manual", trigger=username, started_at=started_at,
any_update_available=True, pull_attempted=True, pull_results=pull_result["results"],
)
except Exception as exc:
logger.exception("Background image pull failed") logger.exception("Background image pull failed")
update_log_service.record_run(
bg_db, run_type="manual", trigger=username, started_at=started_at, error=str(exc),
)
finally: finally:
bg_db.close() bg_db.close()
@@ -241,6 +255,70 @@ async def customers_local_update_status(
return results return results
@router.post("/netbird-updates/apply-all")
async def apply_netbird_client_updates_to_all(
payload: NetbirdClientAutoUpdatePayload,
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
) -> dict[str, Any]:
"""Push a NetBird client automatic-updates version/mode to every customer.
Skips (and reports) customers without a registered API token — they need
a token pasted in via the per-customer endpoint first (older deployments
predating automatic token capture).
"""
if current_user.role != "admin":
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin only.")
deployments = db.query(Deployment).all()
results = []
for dep in deployments:
customer = dep.customer
if not dep.netbird_api_token_encrypted:
results.append({
"customer_id": customer.id, "customer_name": customer.name,
"success": False, "error": "No API token registered.",
})
continue
token = decrypt_value(dep.netbird_api_token_encrypted)
res = await netbird_client_update_service.push_auto_update_settings(
dep.container_prefix, token, payload.version, payload.always
)
results.append({
"customer_id": customer.id, "customer_name": customer.name,
"success": res["ok"], "error": res.get("error"),
})
success_count = sum(1 for r in results if r["success"])
return {
"message": f"Applied to {success_count} of {len(results)} customer(s).",
"updated": success_count,
"results": results,
}
@router.get("/update-logs")
async def get_update_logs(
limit: int = 30,
current_user: User = Depends(get_current_user),
db: Session = Depends(get_db),
) -> list[dict[str, Any]]:
"""Return the most recent NetBird image update runs (scheduled and manual).
Shows exactly what happened on each run: whether an image pull was
attempted and its per-image result, and whether customer containers
were recreated and which ones failed — so this doesn't have to be
reconstructed from container logs.
"""
logs = (
db.query(UpdateRunLog)
.order_by(UpdateRunLog.started_at.desc())
.limit(min(limit, 100))
.all()
)
return [log.to_dict() for log in logs]
@router.post("/customers/update-all") @router.post("/customers/update-all")
async def update_all_customers( async def update_all_customers(
current_user: User = Depends(get_current_user), current_user: User = Depends(get_current_user),
@@ -255,6 +333,7 @@ async def update_all_customers(
if current_user.role != "admin": if current_user.role != "admin":
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin only.") raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin only.")
started_at = datetime.utcnow()
config = db.query(SystemConfig).filter(SystemConfig.id == 1).first() config = db.query(SystemConfig).filter(SystemConfig.id == 1).first()
if not config: if not config:
raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="System not configured.") raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="System not configured.")
@@ -273,6 +352,10 @@ async def update_all_customers(
}) })
if not to_update: if not to_update:
update_log_service.record_run(
db, run_type="manual", trigger=current_user.username,
started_at=started_at, apply_attempted=True, customer_results=[],
)
return {"message": "All customers are already up to date.", "updated": 0, "results": []} return {"message": "All customers are already up to date.", "updated": 0, "results": []}
# Update customers sequentially — one at a time. A failure for one # Update customers sequentially — one at a time. A failure for one
@@ -299,6 +382,10 @@ async def update_all_customers(
}) })
success_count = sum(1 for r in update_results if r["success"]) success_count = sum(1 for r in update_results if r["success"])
update_log_service.record_run(
db, run_type="manual", trigger=current_user.username,
started_at=started_at, apply_attempted=True, customer_results=update_results,
)
return { return {
"message": f"Updated {success_count} of {len(update_results)} customer(s).", "message": f"Updated {success_count} of {len(update_results)} customer(s).",
"updated": success_count, "updated": success_count,
@@ -0,0 +1,129 @@
"""Central control of the NetBird *client* (peer) Automatic Updates feature.
This is the "Settings > Clients > Automatic Updates" toggle inside each
customer's own NetBird dashboard (netbirdio/netbird, added in v0.61.0) — not
to be confused with updating the NetBird Docker images themselves
(app/services/image_service.py).
Talked to over the customer's NetBird Management REST API, authenticated
with a Personal Access Token captured during initial deployment (see
netbird_service.deploy_customer) or pasted in manually for customers
deployed before this feature existed. Requests go over the internal Docker
network directly to the customer's management container — never through
their public dashboard URL.
"""
import json
import logging
from typing import Any
import httpx
logger = logging.getLogger(__name__)
_TIMEOUT = 10
def _base_url(container_prefix: str) -> str:
return f"http://{container_prefix}-management:80"
async def get_current_settings(container_prefix: str, token: str) -> dict[str, Any]:
"""Fetch the customer's current account settings.
Returns:
{"ok": True, "account_id": ..., "settings": {...}} on success, or
{"ok": False, "error": "..."} on failure.
"""
base_url = _base_url(container_prefix)
headers = {"Authorization": f"Token {token}"}
try:
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
resp = await client.get(f"{base_url}/api/accounts", headers=headers)
if resp.status_code != 200:
return {"ok": False, "error": f"GET /api/accounts -> HTTP {resp.status_code}: {resp.text[:300]}"}
accounts = resp.json()
if not accounts:
return {"ok": False, "error": "No account returned by /api/accounts."}
account = accounts[0]
return {"ok": True, "account_id": account["id"], "settings": account.get("settings", {})}
except Exception as exc:
logger.warning("Failed to fetch NetBird account settings for %s: %s", container_prefix, exc)
return {"ok": False, "error": str(exc)}
async def renew_token(container_prefix: str, token: str) -> dict[str, Any]:
"""Mint a fresh 365-day Personal Access Token using the current one.
The old token is left in place (it naturally expires and NetBird gives no
reliable way to identify "our" token among a user's other PATs by content
alone, only by name — which isn't safe to assume is unique). One
harmless unused token lingering until its own expiry is an acceptable
trade-off for not risking deleting a token that turns out to be in use.
"""
base_url = _base_url(container_prefix)
headers = {"Authorization": f"Token {token}"}
try:
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
resp = await client.get(f"{base_url}/api/users", headers=headers)
if resp.status_code != 200:
return {"ok": False, "error": f"GET /api/users -> HTTP {resp.status_code}"}
users = resp.json()
me = next((u for u in users if u.get("is_current")), None)
if not me:
return {"ok": False, "error": "Could not identify current user from /api/users."}
create_resp = await client.post(
f"{base_url}/api/users/{me['id']}/tokens",
headers={**headers, "Content-Type": "application/json"},
content=json.dumps({"name": "MSP Central Management", "expires_in": 365}),
)
if create_resp.status_code not in (200, 201):
return {"ok": False, "error": f"POST tokens -> HTTP {create_resp.status_code}: {create_resp.text[:200]}"}
new_token = create_resp.json().get("plain_token")
if not new_token:
return {"ok": False, "error": "Token creation response had no plain_token."}
verify_resp = await client.get(f"{base_url}/api/accounts", headers={"Authorization": f"Token {new_token}"})
if verify_resp.status_code != 200:
return {"ok": False, "error": "New token failed verification."}
return {"ok": True, "token": new_token}
except Exception as exc:
logger.warning("Failed to renew NetBird API token for %s: %s", container_prefix, exc)
return {"ok": False, "error": str(exc)}
async def push_auto_update_settings(
container_prefix: str, token: str, version: str, always: bool
) -> dict[str, Any]:
"""Set the client automatic-updates version/mode for one customer.
NetBird's account PUT endpoint expects the *entire* settings object, not
a partial patch, so this fetches current settings first and only
overwrites the two auto-update fields.
"""
current = await get_current_settings(container_prefix, token)
if not current["ok"]:
return current
settings = dict(current["settings"])
settings["auto_update_version"] = version
settings["auto_update_always"] = always
base_url = _base_url(container_prefix)
account_id = current["account_id"]
headers = {"Authorization": f"Token {token}", "Content-Type": "application/json"}
body = {"settings": settings}
try:
async with httpx.AsyncClient(timeout=_TIMEOUT) as client:
resp = await client.put(
f"{base_url}/api/accounts/{account_id}", headers=headers, content=json.dumps(body)
)
if resp.status_code != 200:
return {"ok": False, "error": f"PUT /api/accounts/{account_id} -> HTTP {resp.status_code}: {resp.text[:300]}"}
return {"ok": True}
except Exception as exc:
logger.warning("Failed to push NetBird auto-update settings for %s: %s", container_prefix, exc)
return {"ok": False, "error": str(exc)}
+34 -1
View File
@@ -26,6 +26,7 @@ import urllib.error
from datetime import datetime from datetime import datetime
from typing import Any from typing import Any
import httpx
from jinja2 import Environment, FileSystemLoader from jinja2 import Environment, FileSystemLoader
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -231,9 +232,12 @@ async def deploy_customer(db: Session, customer_id: int) -> dict[str, Any]:
"name": customer.name, "name": customer.name,
"email": admin_email, "email": admin_email,
"password": admin_password, "password": admin_password,
"create_pat": True,
"pat_expire_in": 365,
}).encode("utf-8") }).encode("utf-8")
setup_ok = False setup_ok = False
netbird_api_token: str | None = None
for attempt in range(10): for attempt in range(10):
try: try:
req = urllib.request.Request( req = urllib.request.Request(
@@ -245,8 +249,13 @@ async def deploy_customer(db: Session, customer_id: int) -> dict[str, Any]:
with urllib.request.urlopen(req, timeout=10) as resp: with urllib.request.urlopen(req, timeout=10) as resp:
if resp.status in (200, 201): if resp.status in (200, 201):
setup_ok = True setup_ok = True
setup_body = json.loads(resp.read().decode("utf-8"))
netbird_api_token = setup_body.get("personal_access_token")
_log_action(db, customer_id, "deploy", "info", _log_action(db, customer_id, "deploy", "info",
f"Admin user created: {admin_email}") f"Admin user created: {admin_email}"
+ (" (API token captured for central management)"
if netbird_api_token else
" (no API token — NB_SETUP_PAT_ENABLED not active yet on this instance)"))
break break
except urllib.error.HTTPError as e: except urllib.error.HTTPError as e:
body = e.read().decode("utf-8", errors="replace") body = e.read().decode("utf-8", errors="replace")
@@ -340,6 +349,9 @@ async def deploy_customer(db: Session, customer_id: int) -> dict[str, Any]:
deployment.setup_url = setup_url deployment.setup_url = setup_url
deployment.netbird_admin_email = encrypt_value(admin_email) if setup_ok else deployment.netbird_admin_email deployment.netbird_admin_email = encrypt_value(admin_email) if setup_ok else deployment.netbird_admin_email
deployment.netbird_admin_password = encrypt_value(admin_password) if setup_ok else deployment.netbird_admin_password deployment.netbird_admin_password = encrypt_value(admin_password) if setup_ok else deployment.netbird_admin_password
if netbird_api_token:
deployment.netbird_api_token_encrypted = encrypt_value(netbird_api_token)
deployment.netbird_api_token_renewed_at = datetime.utcnow()
deployment.deployment_status = "running" deployment.deployment_status = "running"
deployment.deployed_at = datetime.utcnow() deployment.deployed_at = datetime.utcnow()
else: else:
@@ -354,6 +366,8 @@ async def deploy_customer(db: Session, customer_id: int) -> dict[str, Any]:
setup_url=setup_url, setup_url=setup_url,
netbird_admin_email=encrypt_value(admin_email) if setup_ok else None, netbird_admin_email=encrypt_value(admin_email) if setup_ok else None,
netbird_admin_password=encrypt_value(admin_password) if setup_ok else None, netbird_admin_password=encrypt_value(admin_password) if setup_ok else None,
netbird_api_token_encrypted=encrypt_value(netbird_api_token) if netbird_api_token else None,
netbird_api_token_renewed_at=datetime.utcnow() if netbird_api_token else None,
deployment_status="running", deployment_status="running",
deployed_at=datetime.utcnow(), deployed_at=datetime.utcnow(),
) )
@@ -516,6 +530,25 @@ async def restart_customer(db: Session, customer_id: int) -> dict[str, Any]:
return {"success": ok} return {"success": ok}
async def check_dashboard_reachable(setup_url: str, timeout: float = 10) -> bool:
"""Check whether a customer's NetBird dashboard actually responds over HTTP(S).
A container can be "running" in Docker while the web UI behind it is
still unreachable (e.g. the management service is up but not accepting
connections yet), so this checks the real endpoint rather than the
container state. Any HTTP response (including 4xx/5xx from the app
itself) counts as reachable — only connection failures/timeouts don't.
"""
if not setup_url:
return True # nothing to check (e.g. deployment not fully set up yet)
try:
async with httpx.AsyncClient(timeout=timeout) as client:
await client.get(setup_url)
return True
except httpx.HTTPError:
return False
def get_customer_health(db: Session, customer_id: int) -> dict[str, Any]: def get_customer_health(db: Session, customer_id: int) -> dict[str, Any]:
"""Check health of a customer's deployment.""" """Check health of a customer's deployment."""
deployment = db.query(Deployment).filter(Deployment.customer_id == customer_id).first() deployment = db.query(Deployment).filter(Deployment.customer_id == customer_id).first()
+188 -24
View File
@@ -8,17 +8,30 @@ SystemConfig.auto_update_check_enabled / auto_update_check_time.
import asyncio import asyncio
import logging import logging
from datetime import datetime from datetime import datetime, timedelta
from app.database import SessionLocal from app.database import SessionLocal
from app.models import Deployment, SystemConfig from app.models import Deployment, SystemConfig
from app.services import image_service from app.services import (
docker_service,
image_service,
netbird_client_update_service,
netbird_service,
update_log_service,
)
from app.services.netbird_service import _log_action
from app.utils.security import decrypt_value, encrypt_value
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
_POLL_INTERVAL_SECONDS = 60 _POLL_INTERVAL_SECONDS = 60
_task: asyncio.Task | None = None _task: asyncio.Task | None = None
# NetBird PATs we mint are issued for 365 days; renew well before that so a
# missed tick or a slow rollout never risks the token actually expiring.
_TOKEN_RENEW_AFTER_DAYS = 300
_TOKEN_RENEW_CHECK_HOUR = 4 # run once per day, distinct from the image-check hour
def start() -> None: def start() -> None:
"""Start the background polling task. Safe to call once at app startup.""" """Start the background polling task. Safe to call once at app startup."""
@@ -36,15 +49,68 @@ def stop() -> None:
_task = None _task = None
_last_token_renewal_date = None
async def _poll_loop() -> None: async def _poll_loop() -> None:
while True: while True:
try: try:
await _tick() await _tick()
except Exception: except Exception:
logger.exception("Scheduler tick failed") logger.exception("Scheduler tick failed")
try:
await _token_renewal_tick()
except Exception:
logger.exception("Token renewal tick failed")
try:
await _auto_restart_tick()
except Exception:
logger.exception("Auto-restart tick failed")
await asyncio.sleep(_POLL_INTERVAL_SECONDS) await asyncio.sleep(_POLL_INTERVAL_SECONDS)
async def _token_renewal_tick() -> None:
"""Once a day, renew any NetBird client-update API token nearing its
365-day expiry — keeps central update control working indefinitely
without anyone needing to notice or act.
"""
global _last_token_renewal_date
now = datetime.now()
if now.hour != _TOKEN_RENEW_CHECK_HOUR:
return
if _last_token_renewal_date == now.date():
return
_last_token_renewal_date = now.date()
db = SessionLocal()
try:
cutoff = now - timedelta(days=_TOKEN_RENEW_AFTER_DAYS)
deployments = (
db.query(Deployment)
.filter(Deployment.netbird_api_token_encrypted.isnot(None))
.all()
)
due = [
d for d in deployments
if d.netbird_api_token_renewed_at is None or d.netbird_api_token_renewed_at < cutoff
]
if not due:
return
logger.info("Renewing NetBird API token for %d customer(s)...", len(due))
for d in due:
token = decrypt_value(d.netbird_api_token_encrypted)
result = await netbird_client_update_service.renew_token(d.container_prefix, token)
if result["ok"]:
d.netbird_api_token_encrypted = encrypt_value(result["token"])
d.netbird_api_token_renewed_at = now
db.commit()
logger.info("Renewed NetBird API token for %s.", d.container_prefix)
else:
logger.warning("Failed to renew NetBird API token for %s: %s", d.container_prefix, result.get("error"))
finally:
db.close()
async def _tick() -> None: async def _tick() -> None:
db = SessionLocal() db = SessionLocal()
try: try:
@@ -70,29 +136,49 @@ async def _tick() -> None:
logger.info( logger.info(
"Running scheduled NetBird image update check (auto-apply=%s)...", apply_enabled "Running scheduled NetBird image update check (auto-apply=%s)...", apply_enabled
) )
await _run_check_and_optionally_apply(config, apply_enabled) await _run_check_and_optionally_apply(config, apply_enabled, started_at=now)
finally: finally:
db.close() db.close()
async def _run_check_and_optionally_apply(config: SystemConfig, apply_enabled: bool) -> None: async def _run_check_and_optionally_apply(config: SystemConfig, apply_enabled: bool, started_at: datetime) -> None:
hub_status = await image_service.check_all_images(config) """Check Docker Hub, pull if needed, then (if enabled) recreate any
if not hub_status["any_update_available"]: customer container that isn't running the locally-pulled image yet.
logger.info("Scheduled check: all NetBird images already up to date.")
return
logger.info("Scheduled check: new NetBird image(s) available — pulling.") The "recreate" step is intentionally NOT gated on a Hub update having
pull_result = await image_service.pull_all_images(config) just been found: a customer can already be behind the *locally pulled*
if not pull_result["all_success"]: image from an earlier check/pull that was never applied (e.g. auto-apply
logger.error("Scheduled image pull had failures: %s", pull_result["results"]) was off at the time, or a previous run failed) — that backlog must keep
being cleared on every run, not just on runs that also found something
if not apply_enabled: new on Hub, otherwise it silently never happens.
logger.info("Auto-apply disabled — images pulled, customer containers left untouched.") """
return log_db = SessionLocal()
db = SessionLocal()
try: try:
deployments = db.query(Deployment).all() hub_status = await image_service.check_all_images(config)
any_update = hub_status["any_update_available"]
pull_attempted = False
pull_results = None
if any_update:
logger.info("Scheduled check: new NetBird image(s) available — pulling.")
pull_attempted = True
pull_result = await image_service.pull_all_images(config)
pull_results = pull_result["results"]
if not pull_result["all_success"]:
logger.error("Scheduled image pull had failures: %s", pull_results)
else:
logger.info("Scheduled check: all NetBird images already up to date on Docker Hub.")
if not apply_enabled:
logger.info("Auto-apply disabled — customer containers left untouched.")
update_log_service.record_run(
log_db, run_type="scheduled", trigger="scheduler",
started_at=started_at, any_update_available=any_update,
pull_attempted=pull_attempted, pull_results=pull_results,
)
return
deployments = log_db.query(Deployment).all()
to_update = [] to_update = []
for dep in deployments: for dep in deployments:
cs = image_service.get_customer_container_image_status(dep.container_prefix, config) cs = image_service.get_customer_container_image_status(dep.container_prefix, config)
@@ -104,16 +190,94 @@ async def _run_check_and_optionally_apply(config: SystemConfig, apply_enabled: b
"customer_name": customer.name, "customer_name": customer.name,
}) })
logger.info("Scheduled auto-apply: updating %d customer(s)...", len(to_update)) logger.info("Scheduled auto-apply: updating %d customer(s)...", len(to_update))
customer_results = []
for entry in to_update: for entry in to_update:
try: try:
res = await image_service.update_customer_containers( res = await image_service.update_customer_containers(
entry["instance_dir"], entry["project_name"] entry["instance_dir"], entry["project_name"]
) )
logger.info( ok = res["success"]
"Scheduled update for %s: %s", err = res.get("error")
entry["customer_name"], "OK" if res["success"] else res.get("error"), except Exception as exc:
)
except Exception:
logger.exception("Scheduled update failed for %s", entry["customer_name"]) logger.exception("Scheduled update failed for %s", entry["customer_name"])
ok = False
err = str(exc)
logger.info("Scheduled update for %s: %s", entry["customer_name"], "OK" if ok else err)
customer_results.append({"customer_name": entry["customer_name"], "success": ok, "error": err})
update_log_service.record_run(
log_db, run_type="scheduled", trigger="scheduler",
started_at=started_at, any_update_available=any_update,
pull_attempted=pull_attempted, pull_results=pull_results,
apply_attempted=True, customer_results=customer_results,
)
except Exception as exc:
logger.exception("Scheduled update run failed")
update_log_service.record_run(
log_db, run_type="scheduled", trigger="scheduler",
started_at=started_at, error=str(exc),
)
finally:
log_db.close()
async def _auto_restart_tick() -> None:
"""Every poll interval, restart any customer whose NetBird dashboard
doesn't actually respond over HTTP(S) — a container can be "running" in
Docker while the web UI behind it is still unreachable, which is the
actual symptom customers hit, so this probes the real endpoint rather
than trusting container state alone. Falls back to container status
only if a customer has no setup_url yet. Guarded by
SystemConfig.auto_restart_unreachable_enabled and a per-deployment
cooldown so a persistently broken customer doesn't get restarted in a
tight loop, and skipped for deployments that aren't running/failed
(e.g. intentionally stopped or still pending).
"""
db = SessionLocal()
try:
config = db.query(SystemConfig).filter(SystemConfig.id == 1).first()
if not config or not config.auto_restart_unreachable_enabled:
return
cooldown = timedelta(minutes=config.auto_restart_cooldown_minutes or 30)
now = datetime.now()
deployments = db.query(Deployment).all()
for dep in deployments:
if dep.deployment_status not in ("running", "failed"):
continue
if dep.last_auto_restart_at and now - dep.last_auto_restart_at < cooldown:
continue
reason = None
if dep.setup_url:
if not await netbird_service.check_dashboard_reachable(dep.setup_url):
reason = f"dashboard ({dep.setup_url}) did not respond"
else:
containers = await docker_service.get_container_status_async(dep.container_prefix)
if not containers or not all(c["status"] == "running" for c in containers):
not_running = [c["name"] for c in containers if c["status"] != "running"] or ["no containers found"]
reason = f"container(s) not running: {', '.join(not_running)}"
if not reason:
continue
logger.warning(
"Auto-restart: customer %s unreachable (%s) — restarting.",
dep.container_prefix, reason,
)
_log_action(
db, dep.customer_id, "auto_restart", "info",
f"Automatic restart triggered: {reason}.",
)
try:
result = await netbird_service.restart_customer(db, dep.customer_id)
if not result["success"]:
logger.error("Auto-restart failed for customer %s.", dep.container_prefix)
except Exception:
logger.exception("Auto-restart failed for customer %s.", dep.container_prefix)
dep.last_auto_restart_at = now
db.commit()
finally: finally:
db.close() db.close()
+84
View File
@@ -0,0 +1,84 @@
"""Persists the outcome of every NetBird Docker image update run.
Both the automatic scheduler and the manual "Check/Pull/Update" buttons in
the UI write here, so it's visible afterwards which run pulled which image
successfully, and which customer container recreate failed — instead of
that information only existing in container logs.
"""
import json
import logging
from datetime import datetime
from typing import Any, Optional
from sqlalchemy.orm import Session
from app.models import UpdateRunLog
logger = logging.getLogger(__name__)
def record_run(
db: Session,
*,
run_type: str,
trigger: Optional[str],
started_at: datetime,
any_update_available: Optional[bool] = None,
pull_attempted: bool = False,
pull_results: Optional[dict[str, Any]] = None,
apply_attempted: bool = False,
customer_results: Optional[list[dict[str, Any]]] = None,
error: Optional[str] = None,
) -> UpdateRunLog:
"""Write one completed update-run entry.
Args:
run_type: "scheduled" or "manual".
trigger: username for manual runs, "scheduler" for automatic ones.
pull_results: {image: {"success": bool, "error": str|None}} as returned
by image_service.pull_all_images()["results"].
customer_results: list of {"customer_name", "success", "error"} as
returned by the update-all / update-customer flows.
"""
pull_success = None
if pull_results:
pull_success = all(r.get("success") for r in pull_results.values())
customers_total = len(customer_results) if customer_results is not None else None
customers_updated = (
sum(1 for r in customer_results if r.get("success")) if customer_results is not None else None
)
if error:
status = "failed"
elif pull_attempted and pull_success is False:
status = "failed"
elif apply_attempted and customers_total and customers_updated != customers_total:
status = "partial"
elif not pull_attempted and not customers_total:
# Nothing to pull (Hub already matched local) and no customer
# container was behind the locally pulled image either.
status = "no_update"
else:
status = "success"
log = UpdateRunLog(
run_type=run_type,
trigger=trigger,
started_at=started_at,
finished_at=datetime.utcnow(),
status=status,
any_update_available=any_update_available,
pull_attempted=pull_attempted,
pull_success=pull_success,
pull_details=json.dumps(pull_results) if pull_results is not None else None,
apply_attempted=apply_attempted,
customers_updated=customers_updated,
customers_total=customers_total,
customer_details=json.dumps(customer_results) if customer_results is not None else None,
error=error,
)
db.add(log)
db.commit()
return log
+27
View File
@@ -162,6 +162,12 @@ class SystemConfigUpdate(BaseModel):
auto_update_check_enabled: Optional[bool] = None auto_update_check_enabled: Optional[bool] = None
auto_update_check_time: Optional[str] = Field(None, max_length=5) auto_update_check_time: Optional[str] = Field(None, max_length=5)
auto_update_apply_enabled: Optional[bool] = None auto_update_apply_enabled: Optional[bool] = None
# Master default for the NetBird client (peer) automatic-updates feature
netbird_client_auto_update_version: Optional[str] = Field(None, max_length=50)
netbird_client_auto_update_always: Optional[bool] = None
# Automatic restart of unreachable customer containers
auto_restart_unreachable_enabled: Optional[bool] = None
auto_restart_cooldown_minutes: Optional[int] = Field(None, ge=1, le=1440)
@field_validator("auto_update_check_time") @field_validator("auto_update_check_time")
@classmethod @classmethod
@@ -218,6 +224,27 @@ class SystemConfigUpdate(BaseModel):
return v.lower().strip() return v.lower().strip()
# ---------------------------------------------------------------------------
# NetBird client (peer) automatic updates
# ---------------------------------------------------------------------------
class NetbirdClientAutoUpdatePayload(BaseModel):
"""Push a client auto-update version/mode to one or all customers."""
version: str = Field(..., max_length=50, description="'latest', 'disabled', or a version e.g. '0.61.0'")
always: bool = False
class NetbirdApiTokenPayload(BaseModel):
"""Manually register a NetBird Personal Access Token for a customer.
Needed for customers deployed before automatic PAT capture existed —
create a PAT once in that customer's own NetBird dashboard
(Settings > Service Users / Personal Access Tokens) and paste it here.
"""
token: str = Field(..., min_length=10, max_length=500)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Users # Users
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+9 -9
View File
@@ -5,24 +5,24 @@ sqlalchemy==2.0.23
aiosqlite==0.19.0 aiosqlite==0.19.0
pydantic==2.5.2 pydantic==2.5.2
pydantic-settings==2.1.0 pydantic-settings==2.1.0
python-jose[cryptography]==3.3.0 PyJWT==2.15.1
passlib[bcrypt]==1.7.4 passlib[bcrypt]==1.7.4
bcrypt==4.1.2 bcrypt==4.1.2
cryptography==41.0.7 cryptography==50.0.0
python-multipart==0.0.6 python-multipart==0.0.31
httpx==0.26.0 httpx==0.28.1
jinja2==3.1.2 jinja2==3.1.6
docker==7.0.0 docker==7.0.0
requests<2.32.0 requests<2.32.0
urllib3<2 urllib3<2
psutil==5.9.7 psutil==5.9.7
pyyaml==6.0.1 pyyaml==6.0.1
msal==1.28.0 msal==1.39.0
pyotp==2.9.0 pyotp==2.9.0
qrcode[pil]==7.4.2 qrcode[pil]==7.4.2
slowapi==0.1.9 slowapi==0.1.9
pywinrm>=0.4.3 pywinrm>=0.4.3
ldap3>=2.9.1 ldap3>=2.9.1
pytest==7.4.3 pytest==9.0.3
pytest-asyncio==0.23.2 pytest-asyncio==1.4.0
pytest-httpx==0.28.0 pytest-httpx==0.36.2
+58
View File
@@ -668,6 +668,51 @@
<button type="submit" class="btn btn-primary btn-sm"><i class="bi bi-save me-1"></i><span data-i18n="monitoring.saveAutoUpdateSettings">Save Automation</span></button> <button type="submit" class="btn btn-primary btn-sm"><i class="bi bi-save me-1"></i><span data-i18n="monitoring.saveAutoUpdateSettings">Save Automation</span></button>
</div> </div>
</form> </form>
<hr>
<h6 data-i18n="monitoring.autoRestartTitle">Automatic Restart of Unreachable Containers</h6>
<p class="text-muted small" data-i18n="monitoring.autoRestartHint">Some customers only regain access to their NetBird web UI after a container restart. When enabled, the scheduler periodically checks whether each customer's dashboard actually responds and restarts their containers if it doesn't, logging the action in the customer's log.</p>
<form id="settings-auto-restart-form">
<div class="form-check mb-3">
<input class="form-check-input" type="checkbox" id="cfg-auto-restart-enabled">
<label class="form-check-label" for="cfg-auto-restart-enabled" data-i18n="monitoring.autoRestartEnabled">Automatically restart unreachable customer containers</label>
</div>
<div class="row g-3 align-items-end">
<div class="col-md-3">
<label class="form-label" data-i18n="monitoring.autoRestartCooldown">Cooldown between restarts (minutes)</label>
<input type="number" class="form-control" id="cfg-auto-restart-cooldown" min="1" max="1440" value="30">
</div>
</div>
<div class="mt-3">
<button type="submit" class="btn btn-primary btn-sm"><i class="bi bi-save me-1"></i><span data-i18n="monitoring.saveAutoRestartSettings">Save Automation</span></button>
</div>
</form>
<hr>
<h6 data-i18n="customer.nbuMasterTitle">NetBird Client Auto-Updates (all customers)</h6>
<p class="text-muted small" data-i18n="customer.nbuMasterHint">Controls the "Automatic Updates" setting inside every customer's own NetBird dashboard (Settings &gt; Clients). Set the default here, then push it to all customers at once. Individual customers can still be overridden from their detail page.</p>
<form id="settings-nbu-master-form">
<div class="row g-2 align-items-end">
<div class="col-auto">
<label class="form-label small mb-1" data-i18n="customer.nbuVersion">Client version</label>
<select class="form-select form-select-sm" id="cfg-nbu-version-select" onchange="document.getElementById('cfg-nbu-custom-version').classList.toggle('d-none', this.value !== 'custom')">
<option value="disabled" data-i18n="customer.nbuDisabled">Disabled</option>
<option value="latest" data-i18n="customer.nbuLatest">Latest</option>
<option value="custom" data-i18n="customer.nbuCustom">Specific version</option>
</select>
</div>
<div class="col-auto">
<input type="text" class="form-control form-control-sm d-none" id="cfg-nbu-custom-version" placeholder="0.61.0">
</div>
<div class="col-auto form-check pb-1">
<input class="form-check-input" type="checkbox" id="cfg-nbu-always">
<label class="form-check-label small" for="cfg-nbu-always" data-i18n="customer.nbuForce">Force automatic updates</label>
</div>
</div>
<div class="mt-3">
<button type="submit" class="btn btn-primary btn-sm me-2"><i class="bi bi-save me-1"></i><span data-i18n="customer.nbuSaveDefault">Save Default</span></button>
<button type="button" class="btn btn-outline-warning btn-sm" id="btn-nbu-apply-all" onclick="applyNetbirdUpdatesToAll()"><i class="bi bi-broadcast me-1"></i><span data-i18n="customer.nbuApplyAll">Apply to All Customers</span></button>
</div>
</form>
<div id="nbu-apply-all-result" class="mt-3"></div>
</div> </div>
</div> </div>
</div> </div>
@@ -1233,6 +1278,19 @@
<p class="text-muted mb-0" data-i18n="monitoring.clickCheckUpdates">Click "Check for Updates" to compare local images with Docker Hub.</p> <p class="text-muted mb-0" data-i18n="monitoring.clickCheckUpdates">Click "Check for Updates" to compare local images with Docker Hub.</p>
</div> </div>
</div> </div>
<!-- Update Run Log -->
<div class="card shadow-sm mt-4">
<div class="card-header d-flex justify-content-between align-items-center">
<span><i class="bi bi-clock-history me-2"></i><span data-i18n="monitoring.updateLogTitle">Update Log</span></span>
<button class="btn btn-outline-secondary btn-sm" onclick="loadUpdateLog()" id="btn-refresh-update-log">
<i class="bi bi-arrow-clockwise me-1"></i><span data-i18n="monitoring.refresh">Refresh</span>
</button>
</div>
<div class="card-body" id="update-log-body">
<p class="text-muted mb-0" data-i18n="common.loading">Loading...</p>
</div>
</div>
</div> </div>
</div> </div>
</div> </div>
+318 -1
View File
@@ -571,6 +571,146 @@ function goToPage(page) {
loadCustomers(); loadCustomers();
} }
// ---------------------------------------------------------------------------
// NetBird client (peer) automatic-updates — per-customer
// ---------------------------------------------------------------------------
function _nbuVersionOptions(selected) {
const opts = [
['disabled', t('customer.nbuDisabled')],
['latest', t('customer.nbuLatest')],
['custom', t('customer.nbuCustom')],
];
const isCustom = selected && selected !== 'disabled' && selected !== 'latest';
return opts.map(([v, label]) =>
`<option value="${v}" ${(!isCustom && v === selected) || (isCustom && v === 'custom') ? 'selected' : ''}>${label}</option>`
).join('');
}
async function loadCustomerNetbirdUpdates(id, hasToken) {
const container = document.getElementById('nbu-container');
if (!container) return;
if (!hasToken) {
container.innerHTML = `
<p class="text-muted small mb-2">${t('customer.nbuNoToken')}</p>
<div class="input-group input-group-sm">
<input type="text" class="form-control" id="nbu-token-input" placeholder="${t('customer.nbuTokenPlaceholder')}">
<button class="btn btn-outline-primary" onclick="saveCustomerNetbirdToken(${id})">${t('customer.nbuSaveToken')}</button>
</div>
<div id="nbu-token-result" class="small mt-1"></div>`;
return;
}
container.innerHTML = `<span class="spinner-border spinner-border-sm"></span>`;
try {
const data = await api('GET', `/customers/${id}/netbird-updates`);
const isCustom = data.version && data.version !== 'disabled' && data.version !== 'latest';
let tokenInfoHtml = '';
if (data.token_renewed_at) {
const renewedDate = new Date(data.token_renewed_at);
const expiresDate = new Date(renewedDate);
expiresDate.setDate(expiresDate.getDate() + 365);
tokenInfoHtml = `<div class="small text-muted mb-2">${t('customer.nbuTokenRenewed', { date: renewedDate.toLocaleDateString() })} &middot; ${t('customer.nbuTokenExpires', { date: expiresDate.toLocaleDateString() })}</div>`;
}
container.innerHTML = `
${tokenInfoHtml}
<div class="row g-2 align-items-end">
<div class="col-auto">
<label class="form-label small mb-1">${t('customer.nbuVersion')}</label>
<select class="form-select form-select-sm" id="nbu-version-select" onchange="document.getElementById('nbu-custom-version').classList.toggle('d-none', this.value !== 'custom')">
${_nbuVersionOptions(data.version)}
</select>
</div>
<div class="col-auto">
<input type="text" class="form-control form-control-sm ${isCustom ? '' : 'd-none'}" id="nbu-custom-version" placeholder="0.61.0" value="${isCustom ? esc(data.version) : ''}">
</div>
<div class="col-auto form-check pb-1">
<input class="form-check-input" type="checkbox" id="nbu-always" ${data.always ? 'checked' : ''}>
<label class="form-check-label small" for="nbu-always">${t('customer.nbuForce')}</label>
</div>
<div class="col-auto">
<button class="btn btn-primary btn-sm" onclick="saveCustomerNetbirdUpdate(${id})">${t('customer.nbuSave')}</button>
<button class="btn btn-outline-secondary btn-sm" onclick="syncCustomerNetbirdFromMaster(${id})">${t('customer.nbuSyncMaster')}</button>
</div>
</div>
<div id="nbu-result" class="small mt-2"></div>
<div class="mt-2">
<a href="#" class="small" onclick="event.preventDefault(); showChangeNetbirdToken(${id})">${t('customer.nbuChangeToken')}</a>
</div>
<div id="nbu-change-token-area" class="mt-2 d-none"></div>`;
} catch (err) {
container.innerHTML = `<div class="alert alert-warning py-2 small mb-0">${esc(err.message)}</div>`;
}
}
function showChangeNetbirdToken(id) {
const area = document.getElementById('nbu-change-token-area');
area.classList.remove('d-none');
area.innerHTML = `
<div class="input-group input-group-sm">
<input type="text" class="form-control" id="nbu-token-input" placeholder="${t('customer.nbuTokenPlaceholder')}">
<button class="btn btn-outline-primary" onclick="saveCustomerNetbirdToken(${id})">${t('customer.nbuSaveToken')}</button>
</div>
<div id="nbu-token-result" class="small mt-1"></div>`;
}
async function saveCustomerNetbirdToken(id) {
const input = document.getElementById('nbu-token-input');
const resultEl = document.getElementById('nbu-token-result');
const token = input.value.trim();
if (!token) return;
resultEl.innerHTML = `<span class="spinner-border spinner-border-sm"></span>`;
try {
await api('PUT', `/customers/${id}/netbird-api-token`, { token });
showToast(t('customer.nbuTokenSaved'));
loadCustomerNetbirdUpdates(id, true);
} catch (err) {
resultEl.innerHTML = `<span class="text-danger">${esc(err.message)}</span>`;
}
}
async function _readNbuForm() {
const select = document.getElementById('nbu-version-select').value;
const version = select === 'custom' ? document.getElementById('nbu-custom-version').value.trim() : select;
const always = document.getElementById('nbu-always').checked;
return { version, always };
}
async function saveCustomerNetbirdUpdate(id) {
const resultEl = document.getElementById('nbu-result');
const payload = await _readNbuForm();
if (!payload.version) {
resultEl.innerHTML = `<span class="text-danger">${t('customer.nbuVersionRequired')}</span>`;
return;
}
resultEl.innerHTML = `<span class="spinner-border spinner-border-sm"></span>`;
try {
await api('PUT', `/customers/${id}/netbird-updates`, payload);
showToast(t('customer.nbuSaved'));
loadCustomerNetbirdUpdates(id, true);
} catch (err) {
resultEl.innerHTML = `<span class="text-danger">${esc(err.message)}</span>`;
}
}
async function syncCustomerNetbirdFromMaster(id) {
const resultEl = document.getElementById('nbu-result');
resultEl.innerHTML = `<span class="spinner-border spinner-border-sm"></span>`;
try {
const cfg = await api('GET', '/settings/system');
await api('PUT', `/customers/${id}/netbird-updates`, {
version: cfg.netbird_client_auto_update_version,
always: cfg.netbird_client_auto_update_always,
});
showToast(t('customer.nbuSynced'));
loadCustomerNetbirdUpdates(id, true);
} catch (err) {
resultEl.innerHTML = `<span class="text-danger">${esc(err.message)}</span>`;
}
}
// Search & filter listeners // Search & filter listeners
document.getElementById('search-input').addEventListener('input', debounce(() => { customersPage = 1; loadCustomers(); }, 300)); document.getElementById('search-input').addEventListener('input', debounce(() => { customersPage = 1; loadCustomers(); }, 300));
document.getElementById('status-filter').addEventListener('change', () => { customersPage = 1; loadCustomers(); }); document.getElementById('status-filter').addEventListener('change', () => { customersPage = 1; loadCustomers(); });
@@ -812,6 +952,14 @@ async function viewCustomer(id) {
` : `<p class="text-muted mb-0">${t('customer.credentialsNotAvailable')}</p>`} ` : `<p class="text-muted mb-0">${t('customer.credentialsNotAvailable')}</p>`}
</div> </div>
</div> </div>
<div class="card mt-3">
<div class="card-header">
<strong><i class="bi bi-phone me-1"></i>${t('customer.netbirdClientUpdates')}</strong>
</div>
<div class="card-body" id="nbu-container">
<span class="spinner-border spinner-border-sm"></span>
</div>
</div>
<div class="mt-3"> <div class="mt-3">
<button class="btn btn-success btn-sm me-1" onclick="customerAction(${id},'start')"><i class="bi bi-play-circle me-1"></i>${t('customer.start')}</button> <button class="btn btn-success btn-sm me-1" onclick="customerAction(${id},'start')"><i class="bi bi-play-circle me-1"></i>${t('customer.start')}</button>
<button class="btn btn-warning btn-sm me-1" onclick="customerAction(${id},'stop')"><i class="bi bi-stop-circle me-1"></i>${t('customer.stop')}</button> <button class="btn btn-warning btn-sm me-1" onclick="customerAction(${id},'stop')"><i class="bi bi-stop-circle me-1"></i>${t('customer.stop')}</button>
@@ -824,6 +972,7 @@ async function viewCustomer(id) {
</div> </div>
<div id="detail-update-result"></div> <div id="detail-update-result"></div>
`; `;
loadCustomerNetbirdUpdates(id, d.has_netbird_api_token);
} else { } else {
document.getElementById('detail-deployment-content').innerHTML = ` document.getElementById('detail-deployment-content').innerHTML = `
<p class="text-muted">${t('customer.noDeployment')}</p> <p class="text-muted">${t('customer.noDeployment')}</p>
@@ -947,6 +1096,16 @@ async function loadSettings() {
? new Date(cfg.auto_update_last_run_at).toLocaleString() ? new Date(cfg.auto_update_last_run_at).toLocaleString()
: t('monitoring.autoUpdateNever'); : t('monitoring.autoUpdateNever');
document.getElementById('cfg-auto-restart-enabled').checked = cfg.auto_restart_unreachable_enabled || false;
document.getElementById('cfg-auto-restart-cooldown').value = cfg.auto_restart_cooldown_minutes || 30;
const nbuVersion = cfg.netbird_client_auto_update_version || 'disabled';
const nbuIsCustom = nbuVersion !== 'disabled' && nbuVersion !== 'latest';
document.getElementById('cfg-nbu-version-select').value = nbuIsCustom ? 'custom' : nbuVersion;
document.getElementById('cfg-nbu-custom-version').value = nbuIsCustom ? nbuVersion : '';
document.getElementById('cfg-nbu-custom-version').classList.toggle('d-none', !nbuIsCustom);
document.getElementById('cfg-nbu-always').checked = cfg.netbird_client_auto_update_always || false;
// Branding tab // Branding tab
document.getElementById('cfg-branding-name').value = cfg.branding_name || ''; document.getElementById('cfg-branding-name').value = cfg.branding_name || '';
document.getElementById('cfg-branding-subtitle').value = cfg.branding_subtitle || ''; document.getElementById('cfg-branding-subtitle').value = cfg.branding_subtitle || '';
@@ -1080,6 +1239,81 @@ document.getElementById('settings-auto-update-form').addEventListener('submit',
} }
}); });
// Automatic restart of unreachable customer containers
document.getElementById('settings-auto-restart-form').addEventListener('submit', async (e) => {
e.preventDefault();
try {
await api('PUT', '/settings/system', {
auto_restart_unreachable_enabled: document.getElementById('cfg-auto-restart-enabled').checked,
auto_restart_cooldown_minutes: parseInt(document.getElementById('cfg-auto-restart-cooldown').value, 10) || 30,
});
showSettingsAlert('success', t('messages.imageSettingsSaved'));
} catch (err) {
showSettingsAlert('danger', t('errors.failed', { error: err.message }));
}
});
function _readNbuMasterForm() {
const select = document.getElementById('cfg-nbu-version-select').value;
const version = select === 'custom' ? document.getElementById('cfg-nbu-custom-version').value.trim() : select;
const always = document.getElementById('cfg-nbu-always').checked;
return { version, always };
}
// NetBird client auto-update master default form
document.getElementById('settings-nbu-master-form').addEventListener('submit', async (e) => {
e.preventDefault();
const { version, always } = _readNbuMasterForm();
if (!version) {
showSettingsAlert('danger', t('customer.nbuVersionRequired'));
return;
}
try {
await api('PUT', '/settings/system', {
netbird_client_auto_update_version: version,
netbird_client_auto_update_always: always,
});
showSettingsAlert('success', t('messages.imageSettingsSaved'));
} catch (err) {
showSettingsAlert('danger', t('errors.failed', { error: err.message }));
}
});
async function applyNetbirdUpdatesToAll() {
const { version, always } = _readNbuMasterForm();
if (!version) {
showSettingsAlert('danger', t('customer.nbuVersionRequired'));
return;
}
if (!confirm(t('customer.nbuConfirmApplyAll'))) return;
const btn = document.getElementById('btn-nbu-apply-all');
const resultDiv = document.getElementById('nbu-apply-all-result');
btn.disabled = true;
resultDiv.innerHTML = `<span class="spinner-border spinner-border-sm me-2"></span>${t('common.loading')}`;
try {
const data = await api('POST', '/monitoring/netbird-updates/apply-all', { version, always });
const rows = data.results.map(r => `<tr>
<td>${esc(r.customer_name)}</td>
<td>${r.success
? '<span class="badge bg-success"><i class="bi bi-check-lg"></i> OK</span>'
: '<span class="badge bg-danger"><i class="bi bi-x-lg"></i> Error</span>'}</td>
<td class="small text-muted">${esc(r.error || '')}</td>
</tr>`).join('');
resultDiv.innerHTML = `<div class="alert alert-${data.updated === data.results.length ? 'success' : 'warning'}">
<strong>${esc(data.message)}</strong>
<table class="table table-sm mb-0 mt-2">
<thead><tr><th>${t('monitoring.thName')}</th><th>${t('monitoring.thStatus')}</th><th></th></tr></thead>
<tbody>${rows}</tbody>
</table>
</div>`;
} catch (err) {
resultDiv.innerHTML = `<div class="alert alert-danger">${esc(err.message)}</div>`;
} finally {
btn.disabled = false;
}
}
// Test NPM connection // Test NPM connection
async function testNpmConnection() { async function testNpmConnection() {
const spinner = document.getElementById('npm-test-spinner'); const spinner = document.getElementById('npm-test-spinner');
@@ -1726,7 +1960,90 @@ function togglePasswordVisibility(inputId) {
// Monitoring // Monitoring
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
async function loadMonitoring() { async function loadMonitoring() {
await Promise.all([loadResources(), loadAllCustomerStatuses()]); await Promise.all([loadResources(), loadAllCustomerStatuses(), loadUpdateLog()]);
}
const _updateLogStatusBadge = {
success: 'bg-success',
no_update: 'bg-secondary',
partial: 'bg-warning text-dark',
failed: 'bg-danger',
running: 'bg-info',
};
async function loadUpdateLog() {
const body = document.getElementById('update-log-body');
body.innerHTML = `<span class="spinner-border spinner-border-sm me-2"></span>${t('common.loading')}`;
try {
const logs = await api('GET', '/monitoring/update-logs?limit=30');
if (logs.length === 0) {
body.innerHTML = `<p class="text-muted mb-0">${t('monitoring.noUpdateLogs')}</p>`;
return;
}
const rows = logs.map((l, i) => {
const badgeClass = _updateLogStatusBadge[l.status] || 'bg-secondary';
const typeLabel = l.run_type === 'scheduled' ? t('monitoring.logTypeScheduled') : t('monitoring.logTypeManual');
const pullSummary = l.pull_attempted
? (l.pull_success ? t('monitoring.logPullOk') : t('monitoring.logPullFailed'))
: t('monitoring.logPullSkipped');
const applySummary = l.apply_attempted
? t('monitoring.logApplySummary', { updated: l.customers_updated ?? 0, total: l.customers_total ?? 0 })
: t('monitoring.logApplySkipped');
const detailId = `update-log-detail-${i}`;
const hasDetails = (l.pull_details && Object.keys(l.pull_details).length > 0) ||
(l.customer_details && l.customer_details.length > 0) || l.error;
const detailsHtml = hasDetails ? `
<div class="collapse mt-2" id="${detailId}">
${l.error ? `<div class="alert alert-danger py-2 small mb-2">${esc(l.error)}</div>` : ''}
${l.pull_details ? `
<table class="table table-sm mb-2">
<thead><tr><th>${t('monitoring.thImage')}</th><th>${t('monitoring.thStatus')}</th><th></th></tr></thead>
<tbody>${Object.entries(l.pull_details).map(([img, r]) => `<tr>
<td><code class="small">${esc(img)}</code></td>
<td>${r.success ? `<span class="badge bg-success">OK</span>` : `<span class="badge bg-danger">Error</span>`}</td>
<td class="small text-muted">${esc(r.error || '')}</td>
</tr>`).join('')}</tbody>
</table>` : ''}
${l.customer_details && l.customer_details.length > 0 ? `
<table class="table table-sm mb-0">
<thead><tr><th>${t('monitoring.thName')}</th><th>${t('monitoring.thStatus')}</th><th></th></tr></thead>
<tbody>${l.customer_details.map(r => `<tr>
<td>${esc(r.customer_name)}</td>
<td>${r.success ? `<span class="badge bg-success">OK</span>` : `<span class="badge bg-danger">Error</span>`}</td>
<td class="small text-muted">${esc(r.error || '')}</td>
</tr>`).join('')}</tbody>
</table>` : ''}
</div>` : '';
return `<tr>
<td class="small text-nowrap">${formatDate(l.started_at)}</td>
<td><span class="badge bg-light text-dark border">${typeLabel}</span></td>
<td><span class="badge ${badgeClass}">${t('monitoring.logStatus_' + l.status)}</span></td>
<td class="small">${pullSummary}</td>
<td class="small">${applySummary}</td>
<td class="small text-muted">${esc(l.trigger || '')}</td>
<td>${hasDetails ? `<button class="btn btn-sm btn-outline-secondary" data-bs-toggle="collapse" data-bs-target="#${detailId}"><i class="bi bi-chevron-down"></i></button>` : ''}</td>
</tr>${detailsHtml ? `<tr class="d-none"></tr><tr><td colspan="7" class="p-0 border-0">${detailsHtml}</td></tr>` : ''}`;
}).join('');
body.innerHTML = `
<div class="table-responsive">
<table class="table table-sm mb-0">
<thead class="table-light">
<tr>
<th>${t('monitoring.logTime')}</th>
<th>${t('monitoring.logType')}</th>
<th>${t('monitoring.thStatus')}</th>
<th>${t('monitoring.logPull')}</th>
<th>${t('monitoring.logApply')}</th>
<th>${t('monitoring.logTrigger')}</th>
<th></th>
</tr>
</thead>
<tbody>${rows}</tbody>
</table>
</div>`;
} catch (err) {
body.innerHTML = `<div class="alert alert-danger">${esc(err.message)}</div>`;
}
} }
async function loadResources() { async function loadResources() {
+49 -2
View File
@@ -91,7 +91,30 @@
"lastCheck": "Letzte Prüfung: {time}", "lastCheck": "Letzte Prüfung: {time}",
"openDashboard": "Dashboard öffnen", "openDashboard": "Dashboard öffnen",
"updateImages": "Images aktualisieren", "updateImages": "Images aktualisieren",
"updateInProgress": "Container werden aktualisiert — bitte warten…" "updateInProgress": "Container werden aktualisiert — bitte warten…",
"netbirdClientUpdates": "NetBird Client Auto-Updates",
"nbuNoToken": "Kein API-Token für diesen Kunden hinterlegt. Bei Neu-Deployments wird das automatisch erfasst — für bestehende Kunden einmalig ein Personal Access Token im Kunden-Dashboard erstellen (Settings → Service Users) und hier einfügen.",
"nbuTokenPlaceholder": "Personal Access Token einfügen…",
"nbuSaveToken": "Prüfen & Speichern",
"nbuTokenSaved": "Token gespeichert.",
"nbuChangeToken": "Token ändern",
"nbuTokenRenewed": "Token erneuert am {date}",
"nbuTokenExpires": "gültig bis {date}",
"nbuVersion": "Client-Version",
"nbuDisabled": "Deaktiviert",
"nbuLatest": "Neueste Version",
"nbuCustom": "Bestimmte Version",
"nbuForce": "Automatische Updates erzwingen",
"nbuSave": "Speichern",
"nbuSyncMaster": "Vom Standard übernehmen",
"nbuSaved": "Einstellung übernommen.",
"nbuSynced": "Standard-Einstellung übernommen.",
"nbuVersionRequired": "Bitte eine Version angeben.",
"nbuMasterTitle": "NetBird Client Auto-Updates (alle Kunden)",
"nbuMasterHint": "Steuert die \"Automatische Updates\"-Einstellung im NetBird-Dashboard jedes Kunden (Settings → Clients). Hier den Standard festlegen und auf alle Kunden anwenden. Einzelne Kunden können weiterhin über ihre Detailseite abweichend eingestellt werden.",
"nbuSaveDefault": "Standard speichern",
"nbuApplyAll": "Auf alle Kunden anwenden",
"nbuConfirmApplyAll": "Diese Update-Einstellung auf alle Kunden mit hinterlegtem API-Token anwenden?"
}, },
"settings": { "settings": {
"title": "Systemeinstellungen", "title": "Systemeinstellungen",
@@ -431,6 +454,30 @@
"autoUpdateApplyHint": "Wenn aktiviert, werden nach einer gefundenen Aktualisierung automatisch alle Kunden-Container neu erstellt (kurzer Neustart der Dienste). Wenn deaktiviert, werden nur die Images geladen — die Aktualisierung der Kunden erfolgt weiterhin manuell.", "autoUpdateApplyHint": "Wenn aktiviert, werden nach einer gefundenen Aktualisierung automatisch alle Kunden-Container neu erstellt (kurzer Neustart der Dienste). Wenn deaktiviert, werden nur die Images geladen — die Aktualisierung der Kunden erfolgt weiterhin manuell.",
"autoUpdateLastRun": "Letzte automatische Prüfung", "autoUpdateLastRun": "Letzte automatische Prüfung",
"autoUpdateNever": "Noch nie ausgeführt", "autoUpdateNever": "Noch nie ausgeführt",
"saveAutoUpdateSettings": "Automatisierung speichern" "saveAutoUpdateSettings": "Automatisierung speichern",
"autoRestartTitle": "Automatischer Neustart nicht erreichbarer Container",
"autoRestartHint": "Manche Kunden erhalten den Zugriff auf ihre NetBird-Weboberfläche erst nach einem Container-Neustart zurück. Wenn aktiviert, prüft der Scheduler regelmäßig, ob die Weboberfläche jedes Kunden tatsächlich antwortet, und startet die Container bei Bedarf neu — mit Protokollierung im Kunden-Log.",
"autoRestartEnabled": "Nicht erreichbare Kunden-Container automatisch neu starten",
"autoRestartCooldown": "Wartezeit zwischen Neustarts (Minuten)",
"saveAutoRestartSettings": "Automatisierung speichern",
"updateLogTitle": "Update-Verlauf",
"noUpdateLogs": "Noch keine Update-Läufe protokolliert.",
"logTypeScheduled": "Automatisch",
"logTypeManual": "Manuell",
"logTime": "Zeitpunkt",
"logType": "Typ",
"logPull": "Image-Download",
"logApply": "Kunden-Update",
"logTrigger": "Ausgelöst von",
"logPullOk": "Erfolgreich",
"logPullFailed": "Fehlgeschlagen",
"logPullSkipped": "Nicht nötig",
"logApplySummary": "{updated}/{total} Kunden",
"logApplySkipped": "Nicht ausgeführt",
"logStatus_success": "Erfolgreich",
"logStatus_no_update": "Kein Update nötig",
"logStatus_partial": "Teilweise",
"logStatus_failed": "Fehlgeschlagen",
"logStatus_running": "Läuft…"
} }
} }
+49 -2
View File
@@ -91,7 +91,30 @@
"lastCheck": "Last check: {time}", "lastCheck": "Last check: {time}",
"openDashboard": "Open Dashboard", "openDashboard": "Open Dashboard",
"updateImages": "Update Images", "updateImages": "Update Images",
"updateInProgress": "Updating containers — please wait…" "updateInProgress": "Updating containers — please wait…",
"netbirdClientUpdates": "NetBird Client Auto-Updates",
"nbuNoToken": "No API token registered for this customer. New deployments capture one automatically — for existing customers, create a Personal Access Token once in their dashboard (Settings → Service Users) and paste it here.",
"nbuTokenPlaceholder": "Paste Personal Access Token…",
"nbuSaveToken": "Verify & Save",
"nbuTokenSaved": "Token saved.",
"nbuChangeToken": "Change token",
"nbuTokenRenewed": "Token renewed on {date}",
"nbuTokenExpires": "valid until {date}",
"nbuVersion": "Client version",
"nbuDisabled": "Disabled",
"nbuLatest": "Latest version",
"nbuCustom": "Specific version",
"nbuForce": "Force automatic updates",
"nbuSave": "Save",
"nbuSyncMaster": "Sync from default",
"nbuSaved": "Setting applied.",
"nbuSynced": "Default setting applied.",
"nbuVersionRequired": "Please specify a version.",
"nbuMasterTitle": "NetBird Client Auto-Updates (all customers)",
"nbuMasterHint": "Controls the \"Automatic Updates\" setting inside every customer's own NetBird dashboard (Settings → Clients). Set the default here, then push it to all customers at once. Individual customers can still be overridden from their detail page.",
"nbuSaveDefault": "Save Default",
"nbuApplyAll": "Apply to All Customers",
"nbuConfirmApplyAll": "Apply this update setting to every customer with a registered API token?"
}, },
"customerModal": { "customerModal": {
"newCustomer": "New Customer", "newCustomer": "New Customer",
@@ -338,7 +361,31 @@
"autoUpdateApplyHint": "When enabled, all customer containers are automatically recreated after a new image is found (services briefly restart). When disabled, only the images are pulled — updating customers stays a manual step.", "autoUpdateApplyHint": "When enabled, all customer containers are automatically recreated after a new image is found (services briefly restart). When disabled, only the images are pulled — updating customers stays a manual step.",
"autoUpdateLastRun": "Last automatic check", "autoUpdateLastRun": "Last automatic check",
"autoUpdateNever": "Never run", "autoUpdateNever": "Never run",
"saveAutoUpdateSettings": "Save Automation" "saveAutoUpdateSettings": "Save Automation",
"autoRestartTitle": "Automatic Restart of Unreachable Containers",
"autoRestartHint": "Some customers only regain access to their NetBird web UI after a container restart. When enabled, the scheduler periodically checks whether each customer's dashboard actually responds and restarts their containers if it doesn't, logging the action in the customer's log.",
"autoRestartEnabled": "Automatically restart unreachable customer containers",
"autoRestartCooldown": "Cooldown between restarts (minutes)",
"saveAutoRestartSettings": "Save Automation",
"updateLogTitle": "Update Log",
"noUpdateLogs": "No update runs logged yet.",
"logTypeScheduled": "Automatic",
"logTypeManual": "Manual",
"logTime": "Time",
"logType": "Type",
"logPull": "Image Pull",
"logApply": "Customer Update",
"logTrigger": "Triggered by",
"logPullOk": "Succeeded",
"logPullFailed": "Failed",
"logPullSkipped": "Not needed",
"logApplySummary": "{updated}/{total} customers",
"logApplySkipped": "Not run",
"logStatus_success": "Success",
"logStatus_no_update": "No update needed",
"logStatus_partial": "Partial",
"logStatus_failed": "Failed",
"logStatus_running": "Running…"
}, },
"userModal": { "userModal": {
"title": "New User", "title": "New User",
+5
View File
@@ -20,6 +20,11 @@ services:
image: {{ netbird_management_image }} image: {{ netbird_management_image }}
container_name: netbird-{{ subdomain }}-management container_name: netbird-{{ subdomain }}-management
restart: unless-stopped restart: unless-stopped
environment:
# Allows the MSP appliance to request a Personal Access Token during the
# one-time /api/setup bootstrap call. The endpoint itself locks down
# (412) as soon as the first user exists, so leaving this on is safe.
- NB_SETUP_PAT_ENABLED=true
networks: networks:
- {{ docker_network }} - {{ docker_network }}
volumes: volumes: