Skip to content
NebulaCtrlInstall

Internals

View as Markdown

Rolling vs stop-first for a Laravel service with processes (expertn)

Question: can every process of one NebulaCtrl service (web, worker, scheduler, ssr from one serversideup/php image) deploy "rolling" (new container healthy before the old one stops), or do some need stop-first?

Primary sources only. Pinned versions: laravel/framework 12.x at 18bf64a (v12.70.2), laravel/docs 12.x, serversideup/docker-php at af1462e, inertia-laravel 2.x at 7723950, moby at 9aa9edf. Repo citations are file:line at NebulaCtrl 74d9a62. Anything not verified from a primary source says unverified.

Verdict#

Process Strategy Why Settings to change
web rolling (default) Stateless; the image stops it with SIGQUIT, which nginx/php-fpm treat as graceful. None. 30 s grace is fine.
worker rolling (default) Redis pops are atomic, so no job is claimed twice. The risk is not duplicates but (a) a job killed mid-run when grace < job time and (b) old code briefly pulling jobs. stop_grace_seconds = 75 (was 30). Keep --timeout=60 and retry_after >= 90. If any job sets $timeout above 60, raise grace to $timeout + 15 and retry_after to $timeout + 30.
scheduler stop-first: keep singleton = true schedule:work has no cross-instance guard of its own. Tasks without onOneServer() would run twice during overlap. Add stop_grace_seconds = 10 (was 30). In the app, give every long task withoutOverlapping(<minutes>) with a short expiry.
ssr rolling (default) Stateless HTTP renderer. If it is down or errors, inertia-laravel falls back to client-side rendering instead of failing the request. stop_grace_seconds = 5. Nothing in the process handles SIGQUIT/SIGTERM, so it likely waits out the full grace anyway (see 5).
shared env n/a Migration isolation only works with a cache shared by all containers. Losers of the lock do not wait. Confirm CACHE_STORE is redis (or database), not file/array. Optional, cleaner: run migrate --force as Nebula pre_deploy and set AUTORUN_LARAVEL_MIGRATION=false (see 4).

0. What NebulaCtrl does#

  • A process stops first when its strategy is not rolling, it is a singleton, or the service has volumes: internal/agent/engine/process_deploy.go:497; same rule in internal/services/scaling_view.go:238-243. Plan-level: rolling as soon as any process rolls, internal/deployments/plan.go:61-74.
  • Processes are rolled one at a time, in payload order. Each waits for its own health check before the next starts (process_deploy.go:572-583, 641-701). A stop-first process stops its old container at the start of its own turn (process_deploy.go:665-667, deploy.go:599-612).
  • Old containers of rolling processes are not touched until the commit point after the last process is healthy (process_deploy.go:594-603; comment at 640). So an old worker keeps consuming for the whole deploy, including while the singleton scheduler is stopped and restarted. Only then does retireAll run (process_deploy.go:854-920). It waits DrainDelay (default 2 s, engine.go:73-75,329-331), then stops every old container in parallel, each with its own process's grace (process_deploy.go:884-895,902-918).
  • If a later process fails, the new containers are removed, the previous rolling containers were never touched, and stopped singletons are started again (process_deploy.go:924-940). Docs: docs/operations/processes.md:56-67.
  • Signal. The agent calls Docker's ContainerStop(id, timeout) (internal/agent/docker/sdk.go:414-421) and sets the container StopTimeout to the process's grace (deploy.go:710-728). It sets no signal unless the service's Docker option stopSignal is set (sdk.go:302, dockeropts.go:105-106). Docker then uses the image's STOPSIGNAL, else SIGTERM: docker stop, with the merge of image config into container config at moby daemon/commit.go:116-118 and container.go:596-606. After the grace elapses it sends SIGKILL (daemon/stop.go:56-67). docs/operations/deployments.md:87 says "SIGTERM", which holds only when the image has no STOPSIGNAL.
  • serversideup's fpm-nginx/fpm/fpm-apache images set STOPSIGNAL SIGQUIT (Dockerfile:279-280; docs); the cli image uses SIGTERM (cli.md). Processes share one image and the stop-signal option is service-wide (docs/operations/processes.md:249), so every process of expertn gets SIGQUIT if the image is fpm-nginx (unverified which variant expertn uses). serversideup's own queue/scheduler guides tell you to override to SIGTERM (queue.md:21, task-scheduler.md:28). For queue:work that makes no difference (2). The Dockerfile comment says SIGQUIT was chosen for a graceful nginx/fpm shutdown "instead of S6's preferred SIGTERM", so a service-wide stopSignal = SIGTERM would change how web stops (unverified effect); leave it unset.

1. Is overlap safe, per process?#

web. Yes. Nothing is held in the container; the proxy switches only after the new container is healthy and verified through the proxy (process_deploy.go:743-851). The old one drains for 2 s and then gets a graceful stop. The only overlap issue is old code running against a possibly newer schema (see 4).

worker. No double execution: a worker takes a job with a single Lua script that pops it and moves it to :reserved in one Redis step (LuaScripts::pop, RedisQueue::retrieveNextJob). Two workers cannot receive the same job. What can go wrong:

  • Version skew. Until retire, old workers keep pulling. A job dispatched by new code can run on old worker code (missing class or changed constructor, so unserialize/class-not-found failures), and vice versa. Laravel states workers run the code they started with: "they will not notice changes to your code without being restarted". Keep job payloads backward compatible for one release. If that is not acceptable, the only stop-first switch is singleton = true, which also pins the worker to one replica (scaling_view.go:238-243); the backlog then accumulates for about the grace period and Redis keeps the jobs.
  • Job killed mid-run, if grace is shorter than the job (see 2).

scheduler. Not safe without a guard; see 3.

ssr. Yes; see 5.

2. Worker shutdown, timeouts and grace#

What queue:work does on SIGTERM/SIGQUIT/SIGINT. With pcntl available the worker installs handlers that only set a flag: SIGQUIT, SIGTERM, SIGINT set $shouldQuit = true; SIGUSR2 pauses, SIGCONT resumes (Worker::listenForSignals). The flag is checked between jobs: after runJob returns the loop calls stopIfNecessary, whose shouldQuit branch returns exit 0 (Worker.php:170-250, 358-372). So the worker finishes the current job and then exits cleanly. SIGQUIT (the image's signal) and SIGTERM behave identically. The docs confirm the dependency on signal handling indirectly: "block_for of 0 ... will also prevent signals such as SIGTERM from being handled until the next job has been processed". pcntl is bundled in serversideup images (default-configurations.md:89).

How the four numbers relate.

Is --timeout=60 with grace 30 a problem? Yes, for jobs that can run 30-60 s. SIGQUIT arrives; the worker keeps running the job; at 30 s Docker sends SIGKILL, and the job dies without the SIGALRM path (no failed-job record, no JobTimedOut event). It stays in :reserved and returns after retry_after, with its attempt count already incremented (Lua pop sets attempts+1, LuaScripts.php:78-81). With --tries=3, the third such kill ends as MaxAttemptsExceededException on pickup (markJobAsFailedIfAlreadyExceedsMaxAttempts, Worker.php:592-612). Side effects of the killed run may repeat on the retry. Jobs under about 25 s are unaffected.

Recommended (derivation, not a quote): no job can outlive max(--timeout, job $timeout) because the alarm kills the worker at that point, so grace needs to cover that plus a margin.

  • --timeout=60 (keep), retry_after >= 90 (keep, check expertn's config/queue.php; unverified), stop_grace_seconds = 75.
  • Cost: a busy worker can delay the end of a deploy by up to 75 s, because retire waits for all stops and post_deploy follows (process_deploy.go:900-919). An idle worker exits within its --sleep interval.

3. Scheduler#

Does overlap of two schedule:work run tasks twice? Yes, for any task without a lock. schedule:work fires schedule:run at second 0 of each minute (ScheduleWorkCommand.php:56-78); each instance evaluates due tasks independently. Laravel states: "if the task scheduler is running on three worker servers, the scheduled task will run on all three servers". An overlapping old+new pair is the same situation.

Task kind Protected during overlap? How
->onOneServer() Yes Atomic lock mutexName + 'Hi' (the minute), 3600 s TTL (CacheSchedulingMutex.php:43-57, checked in ScheduleRunCommand.php:131,161). Requires a shared cache: "database, memcached, dynamodb, or redis ... all servers must be communicating with the same central cache server".
->withoutOverlapping() only Partly Per-task cache lock (CacheEventMutex.php:41-52, Event.php:146-149): prevents a task from running while a previous run of it is still going, so two instances will not run it simultaneously, but a run that already finished can be repeated by the other instance within the same minute. Docs: "behind the scenes ... utilizes your application's cache to obtain locks".
Neither No Runs once per scheduler instance. Closures and Schedule::job() (which only dispatches) are included.

Is singleton (stop-first) still right? Yes. It is the only setting that guarantees no overlap regardless of how each task is annotated. Nebula documents this choice: processes.md:42-44,83-85.

Cost of the gap.

  • schedule:work does not catch up. The loop starts schedule:run only when Carbon::now()->second === 0 and that minute was not already run (ScheduleWorkCommand.php:59-66). A task is due only if its cron expression matches the current minute (Event::isDue → expressionPasses, Event.php:272-280). Every minute inside the gap is lost, including a dailyAt() that falls in it. Laravel's docs describe schedule:work as the local-development runner (#running-the-scheduler-locally); serversideup recommends it for containers (task-scheduler.md:9,18).
  • Gap length = old stop + create/start + autorun (DB wait, migrate check, optimize: automations.sh:445-471) + the port-less health period (processes.md:52). Inference: the old stop likely takes the whole grace. ScheduleWorkCommand registers no signal handlers (read in full), PHP is PID 1 (exec "$@", entrypoint:71), and Linux delivers to a container's init only signals it has a handler for (pid_namespaces(7)). Nebula does not enable init by default (engine.go:134, dockeropts.go:105). Not measured. Hence stop_grace_seconds = 10 for the scheduler: shortens the gap by about 20 s at the price of SIGKILLing a task that is mid-run (SIGKILL kills the whole container, children included).
  • Stuck locks after a kill. withoutOverlapping() locks default to 24 h (ManagesAttributes.php:70,145-150) and are released only when the task finishes (Event.php:838-843). A SIGKILL mid-task leaves the lock, and the task is silently skipped until expiry or schedule:clear-cache (docs: "typically only necessary if a task becomes stuck due to an unexpected server problem"). Use withoutOverlapping(<minutes>) with a realistic expiry on every long task. Unverified: which tasks expertn defines.
  • Alternative with no gap: a Nebula cron task running php artisan schedule:run every minute with concurrency = "forbid" (processes.md:83-85,128-134). The same stuck-lock caveat applies if the task is killed.
  • Alternative: roll the scheduler. Only if every task has onOneServer() and the cache is Redis. I do not recommend it without auditing expertn's schedule (unverified).

4. Migrations through serversideup autorun#

What autorun does. The entrypoint runs entrypoint.d/*.sh before exec "$@" for any command, not only the default one (entrypoint:40-71). 50-laravel-automations.sh has no check on the command, so it runs in web, worker, scheduler and ssr (bun) containers (script:47-50,445-471). With isolation on, it runs php artisan migrate --force --isolated, but only if migrate:status succeeds; on a first deploy (no migrations table) it silently omits --isolated (script:122-132). serversideup: "ensure no other containers are running a migration"; "memcached, redis, dynamodb, database, file, or array" cache, same central cache for all servers.

Is --isolated an atomic cache lock? Yes. Laravel: "Laravel will acquire an atomic lock using your application's cache driver before attempting to run your migrations. All other attempts to run the migrate command while that lock is held will not execute; however, the command will still exit with a successful exit status code". Source: MigrateCommand implements Isolatable (line 21); Command::execute takes CacheCommandMutex::create, and on failure prints "already running" and returns the success code without running (Command.php:197-206); the lock is store->lock(...)->get() with a default TTL of one hour (CacheCommandMutex.php:45-60), released in finally (Command.php:217-219).

Processes that lose the lock do not wait and do not fail: they exit 0 from the migrate step and continue to optimize and the real command, possibly while the winner is still altering the schema. Within one Nebula deploy this cannot happen between different processes: they start one after another, each after the previous is healthy (0), and a container is not healthy until its entrypoint scripts finished (the health check runs against the main command, which execs only afterwards; for fpm-nginx the start period is 60 s, Dockerfile:286). The first process to start migrates; the rest find nothing to do. Concurrent starts are only possible for replicas of one process (worker with replicas > 1) or a restart-policy restart. Unverified: whether replicas of a process are created in parallel (createAndStartAll, not read).

Risks to know:

  1. Lock outlives a killed winner. If the container running the migration is SIGKILLed or the deploy is cancelled mid-migration, the finally never runs and the lock remains up to 1 h; the next --isolated run exits 0 without migrating, and the new containers start on an unmigrated schema. Moving migration to Nebula pre_deploy avoids this: it runs once, before any previous container is touched, and a failure aborts the deploy (process_deploy.go:521-526; processes.md:61; the Laravel example at processes.md:111-112). Unverified: whether Nebula runs pre_deploy through the image entrypoint (which would run autorun first as well; harmless with isolation).
  2. Cache not shared. With CACHE_STORE=file or array each container has its own lock store, so isolation protects nothing (the same flag silently "works"). Use redis or database. The scheduler's onOneServer/withoutOverlapping have the same requirement.
  3. Old code on the new schema. Rolling keeps old web and old worker running after the new web has migrated until the commit/retire step (process_deploy.go:594-603). Any non-maintenance deploy has this window; migrations must be backward compatible with the previous release (add before use, drop in the next release). Stop-first would only shrink, not remove, the window for that process. Unverified: expertn's migrations.
  4. Autorun's config:clear / optimize write inside each container's own filesystem (no volumes), so there is nothing shared to race on (script:56-58,168-256).

5. SSR and web#

  • web: see 1. Stateless; port 8080 behind the proxy; 30 s grace with SIGQUIT.
  • ssr: rolling is safe. The Laravel side posts pages to the SSR server over HTTP and, on any exception or empty response, dispatch() returns null, so the page is rendered client-side instead of returning an error (HttpGateway::dispatch, lines 17-41). ssr has a port but no domain, so Nebula does not route it through the proxy (process_deploy.go:199, routedProcess needs both); it is reached by the alias ssr.<service> (docs/operations/networking.md:37), unverified which INERTIA_SSR_URL expertn sets (inertia-laravel's default is http://127.0.0.1:13714, config, which would not reach another container).
  • Asset-version mismatch during overlap. Inertia compares the X-Inertia-Version request header with the server's current version and answers 409 + X-Inertia-Location, which makes the client do a full page visit (protocol; asset versioning). During overlap a browser may hit old web, then new web (or the reverse), producing one extra full reload, not an error. With Vite the version is derived from the manifest automatically. The SSR-specific risk: old web calling the new SSR bundle (or vice versa) can produce server HTML that does not match the client bundle, so hydration warnings until the browser reloads. If old and new ssr containers share the ssr.<service> alias, which one answers during overlap is unverified (I found no primary Docker documentation on alias resolution with several holders). Window is the overlap only (seconds to tens of seconds).
  • Stopping ssr. Inertia's server registers no SIGTERM/SIGQUIT handler; its only shutdown path is the HTTP /shutdown endpoint used by inertia:stop-ssr (server.ts:95,151-153; deployment docs). Whether Bun terminates on SIGQUIT as PID 1 is unverified; if it does not, every deploy waits the full grace on the old ssr container (not user-visible, but it holds up post_deploy). stop_grace_seconds = 5 costs nothing because killing an in-flight render only triggers the client-side fallback above.

Summary of changes for expertn#

toml
[services.expertn.processes.worker]
command            = ["php", "/var/www/html/artisan", "queue:work", "--tries=3", "--timeout=60"]
stop_grace_seconds = 75          # was 30; must exceed max(--timeout, any job $timeout)

[services.expertn.processes.scheduler]
command            = ["php", "/var/www/html/artisan", "schedule:work"]
singleton          = true        # keep: stop-first
stop_grace_seconds = 10          # was 30; shorter gap, tasks must tolerate a kill

[services.expertn.processes.ssr]
command            = ["bun", "/var/www/html/bootstrap/ssr/ssr.js"]
stop_grace_seconds = 5

App-side: retry_after >= 90 on the Redis connection, CACHE_STORE=redis, withoutOverlapping(<minutes>) with a short expiry on long scheduled tasks, backward-compatible migrations and job payloads for one release.