Skip to content
NebulaCtrlInstall

Troubleshooting

View as Markdown

Known limitations

This is a first release. These are the real remaining gaps, not bugs; the list is what the product does not do today. Teams and roles, SSO, passkeys and TOTP two-factor sign-in, git-provider webhooks, notifications, TCP/UDP routes, the web terminal, templates and managed databases all exist: see teams, login methods, git providers, notifications, proxy routes, terminal, templates and databases.

Scope#

  • Compose runs as a stack, with limits. A Compose file can run as one stack service (Compose stacks): recreate-only deploys, one rollback for the whole stack, HTTP routed through domains (published ports are never bound on the host), no build:, no replicas, per-component backups, maintenance mode or tasks, and the safe allow-list unless an instance admin makes a server unrestricted. The alternative is to split a Compose file into services with the one-time Compose import (not kept in sync; build args, bind-mounted directories and the other keys listed there are not carried over). Without the Docker Compose plugin (2.24 or newer) on the server, stacks cannot be deployed.
  • Templates are a curated catalog of 300+. They install as ordinary services on one server (templates); there is no user-facing "save as template" and no runtime import of other catalogs. A newer template revision is offered as an update that stages image, configuration, variable, file and volume changes (your own edits are kept); it never adds or removes services, upgrades a database, changes published ports or deploys by itself, and installs from before updates existed cannot be updated in place. Game-server templates publish TCP/UDP ports on the server's public interface; hosts with a firewall must open them.
  • Managed databases are single-container, with one exception: PostgreSQL can have up to 5 asynchronous read replicas and a manual Promote (database replicas). There is no HA, no automatic failover, no point-in-time recovery, no synchronous or cascading replication, no replicas for the other engines, and no connection pooling or reader proxy; engine settings, init scripts and TLS are not part of the database panel, and dump_restore upgrades (PostGIS, MongoDB) are guided, not automatic (databases). ClickHouse and MongoDB are lightly exercised.
  • A public database port is an IP allow-listed TCP route, not a managed TLS endpoint. It restarts the proxy briefly when the port is new, and the managed databases do not enable TLS on their own (databases).
  • Backups: managed databases (logical dumps), volumes of other services (restic, with a short freeze, snapshot or stop per backups) and the control plane's own database (scheduled, one instance destination; see backup and restore). The secret key is never part of a backup and must be stored separately. Volume backups need the service on one replica (services with volumes always are) and restic-compatible storage. A Docker volume is not a backup.
  • The web terminal is an exec into a replica container; a shell on the server itself is the separate, opt-in host shell (Owners and Admins with two-factor authentication). There is no session recording or replay (by design: I/O is never stored), no docker exec through MCP, no host-shell file transfer and no sharing of a live session between people. The file browser is read-only in production. See web terminal and file browser.
  • Platform services are one of each kind per team (S3 storage, log store, optional Grafana, tailnet control server, single sign-on, container registry), each on one server with the images pinned in the release; the S3 store placed on a server that also runs the databases is not off-site; Loki and Grafana are AGPL-3.0 (unmodified images, separate containers); the identity provider's people are managed by NebulaCtrl and every change redeploys it; changing the public hostname or the control plane's public URL means removing and setting up again; browser sign-in through a real public hostname and Headscale with real Tailscale clients were not exercised (platform services).
  • No Kubernetes, no Swarm, no scheduler: you pick the server for each service. Builds run on the target host by default; an optional build server pushes to a registry and the target pulls by digest (see Build servers: limits: one platform per build, no multi-arch manifests, no ECR/GAR/ACR token refresh, no registry-free transport).
  • Docker Engine only on Linux hosts (Debian 12/13, Ubuntu 22.04, 24.04 and 26.04, amd64/arm64 are the supported targets).
  • OS patching is visibility plus two instance-admin actions (server patching): apt and dnf only, a check at most every 6 hours, no automatic upgrades or reboots, apt-get upgrade semantics (held-back packages are left alone), and the upgrade log is stored when it ends, not streamed.
  • Trusted-team model. Teams have roles (Owner, Admin, Developer, Viewer, plus per-project access), but the Docker socket is root-equivalent on every host (threat model). API tokens have scopes and optional project restrictions.
  • Sign-in: password reset by email needs NEBULACTRL_PUBLIC_URL and an SMTP relay or a team email channel; otherwise an Owner or Admin generates a reset link for the member (forgot password). Accounts that sign in only through an identity provider have no password to reset. Two-factor authentication is an authenticator app or WebAuthn passkeys, with recovery codes; passkeys need HTTPS (or localhost) and a domain name, and are bound to the public hostname.
  • Git sources: generic Git takes HTTPS, HTTP and git:// URLs with a write-only token, and ssh:// / git@host:org/repo.git URLs with a per-service SSH deploy key and a pinned host key (SSH deploy keys). Submodules are checked out recursively (Automatic by default); the repository's token or deploy key is used only for submodules on the same host, other hosts are fetched anonymously over https, and Git LFS and shallow clones are not supported (submodules). A one-off build without cache exists (deploying without cache). A pre-deploy and a post-deploy command exist (tasks); a failing post-deploy command only warns.
  • Production guard: changes an AI client makes to a production environment through MCP are enforced by the server: a person must approve the exact action (MCP elicitation, or a page in the web app) and the check fails closed. REST and CLI tokens are not guarded unless the token was created with "Also ask for approval on REST calls", so CI and deploy hooks keep working; the dashboard's guarded Deploy key is a confirmation in the web app only (the web terminal is the other place the server enforces confirmProduction). Non-production environments never need approval, and a token with "Let MCP clients change production without approval" skips it.
  • Cancel works at any step until the new release is live; after that it is too late and the deployment finishes (canceling). A stop of the previous release in a recreate deployment is not interrupted: the cancel takes effect when it has finished (at most the stop grace period).
  • Log history lives on each server. The agent keeps runtime logs for 7 days (at most 500 MB per service, 5 GB per server; configurable) and queries run there, so history is unavailable while a server is offline, and a service that moves to another server leaves its older lines behind. Drains (Loki, OTLP/HTTP, syslog, HTTP JSON) ship from the agents; there is no control-plane log store, no full-text index and no cross-server log search beyond an environment's merged view. See log history and drains. Request origins by country/city are available; see traffic analytics.
  • Secrets and variables: external secret managers are Vault / OpenBao (KV v2) and Infisical only; Bitwarden Secrets Manager is not supported (its SDK licence forbids redistribution and its protocol is undocumented; revisit if that changes). Shared variables exist at team, project and environment level, not server level (variables, secret managers).
  • Docker options are typed fields, not free-form flags: capabilities to add (safe list) and drop, no-new-privileges, read-only root filesystem, pids limit, hostname (default: the service slug), DNS IPs, curated namespaced sysctls, user and groups, nofile/memlock ulimits, shm size, extra hosts, non-reserved labels, init, stop signal, tmpfs and entrypoint on every server; devices, GPUs, a runtime, privileged mode, the host PID namespace, unconfined seccomp or AppArmor and NET_ADMIN/SYS_ADMIN only on a server an instance admin made unrestricted. Never --ip, host networking on a plain service, restart or log options, the host IPC namespace, host bind mounts or the Docker socket (Docker options). Host kernel parameters such as vm.max_map_count cannot be set from a container: the plan only warns.

Networking#

  • Cross-server networking is the opt-in mesh, not an overlay. Each host has its own Docker networks and its own Traefik proxy. Services in one environment on the same host reach each other by service name (${{service.NEBULA_PRIVATE_HOST}}). Services of one environment on different hosts reach each other through mesh links once the mesh is enabled on both servers: tagged tailnet devices (server nodes and link nodes) that count against your Tailscale plan, TCP only, same environment only (no cross-environment links, no UDP, no Compose stacks), added only by explicit actions, with the ACL generated by NebulaCtrl (managed on Headscale; advisory on Tailscale unless you opt in to a managed push whose token can rewrite the whole policy). Without the mesh, provide a reachable address yourself (VPN, a per-service tailnet, tunnel or public domain). Managed Headscale falls back to Tailscale's public DERP relays (end-to-end encrypted; see mesh). Tailscale SaaS, real NAT traversal, hosts with a firewall and /dev/net/tun inside docker-in-docker were not verified. The managed Cloudflare Tunnel per server carries public HTTP traffic into Traefik only (Cloudflare terminates the public TLS); it does not connect services on different hosts, and it was verified against a fake of the Cloudflare API, not a real account.
  • Network isolation is per environment, opt-in beyond it (networking). Environments, projects and teams on one host no longer share a network, and Block outbound internet cuts a service off (default on for databases), but it is all-or-nothing per service (no per-destination rules), a blocked service cannot join a shared network, shared networks span one server (a mesh link is the only cross-server path, and only within an environment), and a sibling deployed before another began blocking joins the internal network only at its next deploy. Docker networks are not a sandbox for hostile tenants.
  • TCP/UDP exposure goes through proxy routes, which claim a listen port on the server (a clash returns port_in_use). IP rules are an allow list plus a deny list evaluated deny first (rendered as one allow list); there is no further rule ordering, and UDP cannot enforce IP rules at all. Typed proxy options only: there is no raw proxy configuration editor, by design (proxy routes). DNS-01 wildcard certificates work with any lego provider: 19 have a guided form, the rest a generic one; only the guided forms are documented against the lego catalog, and none is verified against a real DNS account (DNS-01 certificates).
  • Hostnames are unique across the whole installation.
  • Per-service tailnet (service tailnet): one tagged device per service (counts against your Tailscale device quota); Headscale gets plain HTTP and raw TCP only (no Serve HTTPS, no Funnel); Funnel bypasses NebulaCtrl's proxy; the web path balances per connection; no egress from the app to tailnet hosts; Tailscale SaaS paths (Serve HTTPS, Funnel, OAuth key minting, device deletion) are not verified against a real tailnet.
  • Service replicas run on one server only (1-10, stateless services; not to be confused with Postgres read replicas, which can live on another server). Multi-host load balancing and automatic failover do not exist; a service runs on one server.

Deployment behavior#

  • No zero-downtime for recreate services: services with volumes, or without an HTTP port, are stopped before the replacement starts. Only volume-less HTTP services use rolling updates.
  • Health semantics follow Coolify: a failed check aborts a deploying release and keeps the old one; it does not kill or restart an already-running container beyond Docker's restart policy. none means "ran for 10 s".
  • Rollback is application image and configuration only; volumes, databases and external services are never rolled back. A git release built on its server can only be rolled back while its built image still exists on the host (default: the newest 5); a release built on a build server is pulled from the registry by digest, as long as the registry still holds it.
  • A git source without a Dockerfile builds with Railpack (0.x, pinned), or a starter Dockerfile if you choose one (builds). Railpack needs outbound access to ghcr.io and either Docker's containerd image store or the docker buildx plugin on the server.
  • Builds run on the target host with its Docker daemon by default (no registry, no build farm): a build competes for resources with running services and untrusted Dockerfiles execute on the host (threat model). With a build server the build moves to a separate host, but that host is a shared privileged builder for every project of the team (per-project BuildKit builders isolate caches, not the kernel), pull-request previews build on their target unless a build server allows previews, and the registry becomes a dependency of those deployments (a deploy falls back to building on the target when no build server or registry is usable).
  • Moving a service to another server does not move its volumes.
  • Clone and move copy configuration, not data. A cloned service or environment gets new, empty volumes and new, empty databases (restore a dump into them); route services, Compose stacks and read replicas are not cloned, and services that are in a replica group or have cross-server links cannot be moved (clone and move).
  • Replacing a service's container closes WebSocket connections still open to the old container: it is stopped after the 2 s drain and its stop grace period. A route change itself does not close existing connections (Traefik hot-reloads its configuration).

Platform#

  • One control-plane instance (no high availability); it keeps agent connections in memory. Agents reconnect and reconcile after a restart, and workloads keep running, but there is no management while it is down.
  • Agents upgrade themselves to the control plane's version (automatic, or on request, with a rollback when the new binary does not start), but only from an agent that already has the feature: an older agent is upgraded once by re-running the installer. The control plane is upgraded by re-running the installer; there is no dashboard "update" button (upgrades).
  • Forward-only migrations: downgrading the control plane means restoring a backup.
  • One secret key serves the whole instance (every team); there is no envelope encryption, KMS or TPM sealing of it. Rotation runs on a live instance but is manual: secrets at rest.
  • Metrics and traffic analytics retention is per server and bounded (monitoring); there is no long-term time-series store.
  • The control plane's public image is not published yet; Compose builds from source.
  • NebulaCtrl is AGPL-3.0-only (ADR 0005). Anyone who runs a modified version as a network service must offer its users the modified source.