Draining servers and the fleet overview

When to drain#

Drain a server to empty it before maintenance (kernel update, reboot, disk work) or before retiring a VPS. A drain moves every service placed on the server to other servers, one at a time, and closes the server to new placements.

Draining is not required to revoke a server, but revoking leaves the containers running and the services still placed on it (Servers and agents). Drain first, then revoke.

Volumes are not copied. A service with volumes, and any service of kind database, starts with empty volumes on its target. Move the data yourself before you drain (see Stateful services).

In the UI: open the server's detail page and use the drain sheet (the final button is hold-to-confirm: "Hold to drain <server>"). The same flow is available over the API.

Plan (dry run)#

POST /api/v1/servers/{serverId}/drain/plan changes nothing. It returns one move per service on the server, plus a capacity verdict. The body is optional: {"targetServerId": "<id>"} sends every service to that server; omitting it picks a target per service.

Move order: stateless services first, services with volumes or of kind database last. A failure stops the drain, so data-bearing services are touched last.

Automatic target#

Without targetServerId, each service goes to the eligible server with the most free memory. A server is eligible when it is online, not the source, not revoked, and its drain state is neither draining nor drained.

Free memory (headroom) of a candidate:

total memory − max(memory used, committed limits) − memory already assigned by this plan

• Committed limits is the sum of the memory limits (memoryMb) of the services already placed on the candidate.
• Assigned is what earlier moves of the same plan already booked on it, so one big server does not receive everything unless it still has the most room.
• A service's demand is its memory limit; with no limit, its latest usage sample (at most 2 minutes old); with neither, nothing is booked.
• Ties are broken by server name, then ID.
• A server that has not reported its memory ranks below every server that has. If it is the only candidate, the plan still picks it and warns that capacity is unknown.

With an explicit target the server must exist, differ from the source, not be revoked, and not be draining or drained (422 server_draining).

What the plan reports#

Per move: targetServerId/targetServerName, stateful, warnings[], blocking[]. Overall: capacity.targetFits and capacity.reasons[].

Blocking problems (the drain cannot start):

Capacity: targetFits is false when a target would exceed its memory after the assigned moves; reasons say by how much. If a target has not reported its memory the check is skipped for it and a reason says so. The start endpoint does not re-check targetFits; read it before you start.

Warnings the plan can emit:

• the target was chosen automatically (and how much memory remains free, or that capacity is unknown);
• volumes are not copied / a database starts empty on the target;
• the service has never been deployed: only its placement changes, nothing is started;
• the service has staged changes: they are deployed together with the move;
• the service has no memory limit (and possibly no usage data): the capacity check approximates or cannot account for it;
• a deployment of the service is in progress: the drain waits for it to finish;
• the service has domains and the target has a different public IP: point the A records to the target after the move;
• the source server is not online: the old container cannot be stopped until its agent reconnects.

Stateful services#

A service is stateful when it has at least one volume or its kind is database. On the target it starts with empty volumes.

1. Back up or copy the data to the target host yourself.
2. Start the drain with "confirmStateful": true. Without it the API answers 422 confirmation_required naming the stateful services. In the UI this is the checkbox "I have moved the data, or I accept losing it."

Nothing in the drain code copies or deletes volume data.

Starting the drain#

POST /api/v1/servers/{serverId}/drain with:

{
  "moves": [{"serviceId": "…", "targetServerId": "…"}],
  "confirmStateful": true
}

Rules enforced on start:

• Every service on the server appears in moves exactly once (422 incomplete_moves lists the missing ones; 422 invalid_moves for duplicates or services not on this server).
• Every target passes the checks above and is online (422 target_offline otherwise). Plan blockers (no_target) are returned as errors.
• 409 drain_in_progress if the server is already draining.
• Idempotency-Key is honored.

The call returns 202. In one transaction the server's drain state becomes draining, a drain run is stored, a background job is enqueued, and the audit event server.drain.start is written (including the plan and confirmStateful).

How the moves run#

Moves run sequentially, in plan order. Per service:

1. If a deployment of the service is active, wait (3 s polls) until it ends.
2. Change the service's placement to the target. This is a normal service update, so it increments the service generation and is audited under the initiator.
3. If the service was never deployed, it is marked done ("Not deployed yet: only the placement changed.").
4. Otherwise create a deployment with trigger drain for the release content that is running now (a git service is pinned to its current commit SHA; a drain never ships new code). Staged config and variable changes are deployed with it.
5. Wait until that deployment ends. succeeded marks the move done. Per the endpoint description, the old container is removed once the new one is live.

A service that was deleted during the drain is skipped as done. A drain job interrupted by a control-plane restart resumes from the stored progress and adopts an already running drain deployment instead of starting a second one.

The job acts as the initiator with projects:read, projects:write and deployments:write, so deployments and audit entries are attributed to them.

When every move is done, the run is completed, the server's drain state becomes drained, and server.drain.complete is audited.

Follow progress with GET /api/v1/servers/{serverId}/drain (the running drain, else the newest finished one, else {"drain": null}). Each move has the status pending, moving, deploying, done or failed, and deploymentStatus live from its deployment.

Failure semantics#

The first move that fails stops the drain:

• Failure means the placement update was rejected, or the drain deployment ended in anything but succeeded (failed, rolled_back, rollback_failed, canceled, or the deployment disappeared).
• That service's placement is reverted to the source server, so the placement matches what is actually running. If its placement or generation was changed meanwhile, the revert is skipped and the move's message says "the service's placement was changed meanwhile and was not reverted".
• Services not yet started are untouched and keep running on the source. Services already moved stay on their targets; nothing is moved back.
• The run is failed; the server's drain state is failed, with drainMessage = Moving <service> to <server> failed: <reason>. server.drain.fail is audited.

In state failed the server accepts new placements again (the scheduling guard applies only to draining and drained). Fix the cause, then either Retry (re-plan) in the UI, which plans the services that are left, or undrain.

Cancel#

POST /api/v1/servers/{serverId}/drain/cancel asks the drain to stop after the move in flight. A move that is moving or deploying is finished first; no further move starts. Nothing is rolled back.

409 no_drain_in_progress when no drain is running. server.drain.cancel is audited.

Undrain#

POST /api/v1/servers/{serverId}/undrain sets a drained or failed server back to none so it can receive services again. Services that were moved stay where they are.

• While a drain is running: 409 drain_in_progress (cancel first).
• On a server whose state is already none: no change, the server is returned.

Scheduling guard#

A server in drain state draining or drained refuses new placements: creating a service on it, or changing a service's placement to it, returns 422 server_draining ("<name> is being drained"). Editing a service that stays on the server is not affected. The same guard rejects such a server as an explicit drain target. The UI greys these servers out in the service placement select and in the drain target list.

Drain states: none, draining, drained, failed. drainState and drainMessage appear on the fleet overview; the drain run itself carries createdAt/finishedAt.

Required scopes#

The drain touches every project placed on the server, so tokens restricted to specific projects are refused (403) on every drain endpoint. Session users hold every scope.

API endpoints#

All under /api/v1, tag fleet:

Deployments created by a drain have trigger drain.

Fleet overview#

GET /api/v1/fleet (UI: the Fleet view on the Servers page) lists every server of the workspace with:

• status, drainState, drainMessage;
• cpus, memTotalBytes, diskTotalBytes, and the latest cpuPercent, memUsedBytes, diskUsedBytes from heartbeats;
• committedMemBytes: the sum of the memory limits of the services placed on the server;
• unlimitedServices: how many placed services have no memory limit (they are not in the committed sum);
• overcommitted: true when committedMemBytes is greater than memTotalBytes (never when the total is unknown; equal is not overcommit);
• services[]: name, project, environment, kind, memLimitBytes (0 = unlimited), latest usage, stateful, and a status (undeployed, deploying, running, unhealthy, stopped, missing, failed, unknown).

totals sums servers, services, memory total, used and committed.

Server numbers always cover all services. The services[] rows are limited to the projects the caller may see.

Overcommit is a warning, not a limit: the code only flags it. The drain planner uses max(used, committed) in its headroom formula, so it steers moves away from overcommitted servers.