Skip to main content
Use this guide after installing Roomote and completing your first task. These commands apply to deployments created with the Linux installer. For a hosting platform, follow its deployment guide for lifecycle operations. Run host commands on the Linux machine running Roomote. Use sudo if your account needs it to access the deployment.

Day-2 operations

The installer also sets up the roomote host CLI for common operations:
roomote backup prompts for a passphrase and writes a versioned .roomote bundle under /opt/roomote/backups. The bundle contains PostgreSQL, the deployment configuration and encryption/signing keys, local MinIO artifacts, schema metadata, the exact deployed image identities, and both the Memory volume and isolated Memory database when the brain profile is enabled. Store the passphrase separately in your secret manager; the backup cannot be restored without it. Webhook and delivery lifecycle entries in roomote logs use single-line JSON with an event field plus safe correlation fields such as provider, delivery, session, task, run, and job IDs. Search these entries to follow Telegram messages and GitHub pull-request review activity from receipt through persistence, dispatch, and final delivery. Message bodies, raw payloads, credentials, signatures, and token-bearing URLs are excluded. Use roomote backup --include-redis when queued work, BullMQ schedules, sessions, and other transient Redis state must survive. Backups briefly stop application writers (and active Docker task workers) so the included stores share a documented consistency point. If object storage is external, the bundle records its endpoint and bucket but does not copy its objects; keep a provider-level backup of that bucket. Roomote exposes /health/bullmq for queue-processing health. It returns healthy only when the background worker has recently processed its heartbeat job and no session event has remained queued for more than five minutes. Any overdue event makes the check fail; authenticated error responses report the overdue count, while unauthenticated responses omit diagnostics. Use it alongside container liveness and roomote logs: a stale heartbeat or growing backlog can reveal a worker process that is still running but no longer draining jobs. Restore only after installing Roomote on the replacement host:
Restore verifies the encrypted bundle and checksums before replacing any state, restores the original .env (including ENCRYPTION_KEY), repopulates empty PostgreSQL/MinIO/Redis volumes, and starts the recorded Roomote release.

Upgrades and rollback

roomote upgrade is designed so a bad release cannot strand your deployment:
  • A backup comes first. Every upgrade creates an encrypted pre-upgrade bundle under /opt/roomote/backups before anything changes. Pass a passphrase with --backup-passphrase-file (or ROOMOTE_BACKUP_PASSPHRASE); otherwise one is generated and stored next to the bundle. Use --skip-backup to opt out.
  • Migrations run before services are replaced. Database migrations apply in a single transaction while the previous release keeps serving. The bundled migration runner makes up to three total attempts after a transient Postgres connection failure: the initial run plus at most two retries. If migration still fails, the schema rolls back, the previous configuration is restored, and the previous release stays up.
  • Rollback is one command. Every release’s schema keeps the previous release working, so roomote rollback re-deploys the prior release without touching the database. roomote upgrade <tag> does the same for any retained tag, and restoring the pre-upgrade bundle is the last-resort path that also rewinds data.
Use roomote upgrade instead of updating application image references alone. The command refreshes the release’s Compose and Caddy configuration together; mixing newer application images with an older Caddyfile can leave routes used by the new controller unavailable until the deployment configuration is also updated. The supported rollback target is the release immediately before the current one. Check the running application version and applied schema migration at any time under Settings → Deployment → Diagnostics; both are also recorded in every backup bundle’s manifest. When upgrading from Roomote 1.19.5 or earlier to 1.19.6 or later, upgrade all servers handling sign-in and account linking together. Restart any sign-in or account-linking attempts begun before the upgrade. Existing signed-in sessions are retained, and registered OAuth callback URLs do not need to change. Roomote surfaces release history and available updates in the web app:
  • Admins on self-hosted deployments see an update notice in the sidenav when a newer GitHub release is available. The update dialog keeps that target and its GitHub link visible even when the running image does not yet contain the newer release’s notes. Use roomote upgrade (or your image roll-forward process) to install it.
  • Everyone can open About Roomote and select See all Roomote releases to browse the current and previous release notes bundled in the running image, with the latest release expanded. After an upgrade, Roomote also shows the latest “what’s new” notice once.

HTTP telemetry after upgrading to Roomote 1.21.0

Roomote 1.21.0 includes an OpenTelemetry HTTP instrumentation security update that redacts sensitive signed-URL query parameters in incoming request spans. The updated instrumentation emits only stable HTTP semantic conventions; legacy and duplicate HTTP attributes and metrics are removed, regardless of OTEL_SEMCONV_STABILITY_OPT_IN. If you collect HTTP telemetry for custom dashboards or alerts, review your collector processing and queries before upgrading. Update references to legacy fields and metrics to the stable conventions, and verify that request counts, latency charts, and error alerts still receive data after the upgrade. See OpenTelemetry’s HTTP semantic-convention migration guide for the field, metric, and unit changes. This affects instrumented HTTP signals, not Roomote’s in-app Cost Analytics.

Worker runtime and saved snapshots

The worker image ships Node 22.23.4, and the TLS trust minimum is Node 22.23.4, 24.22.1 or 26.12.1 (or a newer patch within those branches). When an older image or saved snapshot starts, worker bootstrap logs a warning and continues. Starting with Roomote 1.19.1, tasks skip environment snapshots built by another release, or with no recorded release, and run environment setup from scratch instead of failing during worker startup. After an upgrade, initial task starts can therefore take longer. Roomote checks for stale environment snapshots every five minutes and queues replacements from the current image, with rebuilds paced across environments; the daily refresh remains in place too. Rebuild custom worker images yourself, and migrate or update saved task snapshots you intend to keep using. The fresh-start fallback applies to environment snapshots, not saved task snapshots. Starting with Roomote 1.19.2, follow-ups can resume saved task snapshots created with an older release: worker startup warns about the older image and continues, preserving the task’s files. The installed worker must still match the application release. This does not update tools inside the snapshot. An application upgrade does not replace tools or certificate bundles inside an existing snapshot. Update its OS certificates, Python environments and independent harness binaries too, and recreate warm sandboxes. A successful browser or curl check does not verify every application’s TLS trust store. The trust updates in Roomote 1.19.0 are partial remediation, not a fleet-wide TLS guarantee. Bun/Brain/OpenCode, MinIO/mc, and other independent verifiers require separate updates and verification. Environment snapshots pick up the updated runtime when rebuilt, including the automatic post-upgrade rebuilds in Roomote 1.19.1 and later. Rebuild custom worker images, migrate saved task snapshots you keep, and restart affected running tools; upgrading the application alone does not update them.

Rotate the encryption key

Encryption-key rotation requires Roomote 1.19.0 or later on every service. Before changing any keys, upgrade all services to images that include this feature. Older releases, including v1.18.2, do not support ENCRYPTION_KEY_PREVIOUS or the reencrypt-secrets command.
ENCRYPTION_KEY encrypts the secrets Roomote stores in PostgreSQL: integration credentials and OAuth tokens, environment variables, API keys, webhook secrets, and generated auth keypairs. Changing it alone makes every stored secret unreadable. Rotate it in two phases instead, keeping the old key available until every stored value has been re-encrypted.
  1. Take a backup. Run roomote backup so you can return to the old key and data if anything goes wrong.
  2. Decide what signs sign-in sessions. If BETTER_AUTH_SECRET is unset, sessions are signed with the encryption key. During the rotation Roomote keeps signing them with ENCRYPTION_KEY_PREVIOUS, so users stay signed in, but removing the previous key in step 7 would switch session signing to the new key and sign everyone out. To avoid that, set BETTER_AUTH_SECRET in /opt/roomote/.env now: to the old ENCRYPTION_KEY value to keep existing sessions, or to a new random value (openssl rand -base64 32) if you are rotating because the old key may be exposed. A new value signs everyone out once. The Compose file passes it to the web service, which signs sessions; it takes effect with the roomote up in the next step.
  3. Install the new key. In /opt/roomote/.env, move the current value to ENCRYPTION_KEY_PREVIOUS and set a new ENCRYPTION_KEY:
    Both must be at least 32 characters. Apply them with roomote up, which recreates the services with the new environment. Roomote now writes every new or updated secret with the new key and reads older ones with the previous key.
  4. Preview the re-encryption. From /opt/roomote:
    The command lists every table and column that holds encrypted values with counts of rows already on the current key (current), rows with values that would be rewritten (migrated), rows with values that open with neither key (unreadable), rows that are not encrypted at all (not_encrypted), and rows still holding a previous-key value (remaining_on_previous). A row that stores several secrets, such as a custom MCP server’s headers, can count as both migrated and unreadable. Nothing is written.
  5. Re-encrypt. Run the same command without --dry-run. It works in batches, rewrites only values that still use the previous key (leaving unreadable values in the same row untouched), and never overwrites a value that changes while it runs; such rows are reported as conflicts. It is safe to run again at any time.
  6. Verify. Run the dry run again. When it reports No rows remain on the previous key, the rotation is complete. The command exits non-zero while any row is unreadable or conflicted; re-run it for conflicts. Unreadable rows were already unreadable before the rotation (they were written with some other key); re-save those secrets in Roomote.
  7. Remove the previous key. Delete ENCRYPTION_KEY_PREVIOUS from .env and run roomote up again. Keep the old key in your secret manager until you no longer need backups taken before the rotation, since those backups still contain data encrypted with it.
On a hosting platform, set ENCRYPTION_KEY_PREVIOUS and the new ENCRYPTION_KEY on every service that has ENCRYPTION_KEY (web, api, bullmq, and controller), set BETTER_AUTH_SECRET on the web service, redeploy, and run /roomote/.docker/app/entrypoint.sh reencrypt-secrets --dry-run from a shell in the api service. The previous key is never passed to task sandboxes. The encryption key also keys a few non-secret identifiers. Credential substitutes issued to runs that are in progress during step 3 stop matching until the run restarts or resumes, and provider usage-limit alerts may repeat once because their credential fingerprints change.

Common issues

  • Callbacks fail. For providers that use callbacks or webhooks, confirm the deployment has a public HTTPS URL and that the provider is using that exact URL. Discord uses its Gateway service instead of a public callback.
  • The first task cannot clone a repository. Workers authenticate git with a short-lived GitHub App installation token created when the run starts — they do not read a GH_TOKEN/GITHUB_TOKEN from the container environment. If a task fails with “No GitHub credentials are available”, verify the GitHub App is installed for the repository owner, the installation covers the repository, and the repository appears in the source-control settings after a sync.
  • The agent cannot run useful commands. Add missing services, environment variables, setup commands, or tool versions to the environment.
  • Chat messages do not reach Roomote. Confirm the app is installed, invited to the channel, and using the current callback URL.
  • Roomote reports that local working storage is full. Clean up container storage, then recreate the affected application containers. Installer-managed Compose deployments can raise ROOMOTE_APP_TMPFS_SIZE above its 512m default before recreating them; keep the limit within the host’s available memory.
  • Memory climbs for hours, or a service is OOM-killed under a container memory limit. Node sizes its heap from the memory it can see (the host’s, not the container’s), so it defers garbage collection and lets memory drift well above what the service needs; a container memory cap then kills the process before Node ever feels pressure. Current app images cap each Node service’s heap by default (768 MB for web and api, 512 MB for controller and bullmq, reduced to ~75% of the container’s memory when a cgroup limit is set). Set NODE_OPTIONS=--max-old-space-size=<MB> on a service to override the default, or on older images to add the cap manually.

Decision model on CPU (optional)

Roomote asks a decision model small typed questions: whether a turn is worth remembering, who a thread reply is for, when a running task has news for the user, which model a delegated task should use. With a TypeSafe key, Jev answers them. Without one, the judgment profile runs a CPU decision model beside the stack, so these features work with no external decision API and no GPU. Add these to .env:
The sidecar serves roomote/roomote-judgment-gliner, a GLiNER 2.5 model fine-tuned on Roomote’s decisions and trained only on synthetic data. It downloads the model on first start and keeps it in the roomote_judgment_models volume. Its model card lists the decisions it covers and its accuracy; on real decisions it trails Jev by a few points, and Roomote acts on a decision only when the model is confident and the decision is in its evaluated policy. Unsupported decisions keep their existing safe fallback. To serve a different checkpoint, set JUDGMENT_MODEL to a Hugging Face id or path (and JUDGMENT_HF_TOKEN if it is private). The sidecar is reachable only on the stack’s internal network; set R_JUDGMENT_UPSTREAM_API_KEY as well to require a bearer token. Sizing: the container is capped at JUDGMENT_MEMORY_LIMIT (default 4g) and JUDGMENT_CPUS (default 4). It uses about 1.5 GB, and on 4 cores a decision takes about 0.3 to 1 second (p90 about 2 seconds). Decisions run one at a time, so a burst of them queues and more cores make each one faster. The production install (install.sh) and the Railway, Render, and Coolify templates carry the same service, opt-in like Memory: in the production install it is the judgment profile, and in the templates it idles without loading its model until R_JUDGMENT_UPSTREAM_URL points at it and the Roomote judgment model is selected. Each template’s README has the steps.