coder/q5vault/operations/verification.md

title, description, source, reviewed

title	description	source	reviewed
✅ Verification	Numbered proof checklist for the Coder stack with exact commands and expected output, including the deliberate contrast against the Authelia-gated cdsr.loca.zone host.	verified deployment on the loca operator host, 2026-08-19; user, template, workspace and app checks verified 2026-08-20	2026-08-20

✅ Verification

Run this list in order after any change to the compose file, the nginx vhost, the certificate or the image pin. Items 1 to 6 were observed on 2026-08-19 with the outputs shown; items 8, 9 and 10 were observed on 2026-08-20 after the first-user bootstrap, the template push and the first workspace build, and item 11 was re-measured the same day. Item 7 becomes true again on every wiki build.

1️⃣ Both containers up, database healthy

cd /home/loca/dev/coder && sudo -n docker compose ps

• expect container coder with state running and the single port mapping 127.0.0.1:52226->7080/tcp
• expect container coder-database with state running and health healthy, and no port mapping at all
• a coder-database stuck in starting means the pg_isready -U coder -d coder healthcheck has not passed yet; coder will not start until it does

2️⃣ Server health on loopback

curl -sf http://127.0.0.1:52226/healthz

• expect exactly OK
• a non-zero exit from curl -sf means the server is not answering; check sudo -n docker compose logs --tail 200 coder

3️⃣ Running build matches the pin

curl -sf http://127.0.0.1:52226/api/v2/buildinfo

• expect version v2.35.3+65e2bfb, matching config/runtime-version.txt (v2.35.3) and the image tag in docker-compose.yml
• expect dashboard_url https://code.loca.zone, which proves CODER_ACCESS_URL reached the server
• expect telemetry false, which proves CODER_TELEMETRY_ENABLE took effect
• no jq on this host; if the raw JSON needs shaping, use python3 -c with the json module

4️⃣ Public app host answers directly, with no auth redirect

curl -sSI https://code.loca.zone/

• expect HTTP/1.1 200 OK (or HTTP/2 200), served by the Coder SPA, with no Location header
• expect the security headers from snippets/security-headers-app.conf
• a 302 to https://auth.loca.zone/... here is a regression: someone added an Authelia rule for code.loca.zone. Remove it, reload Authelia, and re-read operations/auth-model

5️⃣ Contrast check against the gated sibling host

curl -sSI https://cdsr.loca.zone/

• expect HTTP/1.1 302 with a Location pointing at https://auth.loca.zone/, because cdsr.loca.zone carries an explicit one_factor Authelia rule
• this is the control for item 4: the same nginx and the same Authelia produce a redirect there and a direct 200 here, which proves the difference is the per-host rule and not a broken gate
• if both hosts return 200, Authelia itself is down or its rule set was truncated; if both return 302, the Coder vhost was gated by mistake

6️⃣ Certificate lineage covers both names

sudo -n certbot certificates

• expect one lineage named code.loca.zone whose domains are code.loca.zone and wiki.code.loca.zone, expiring 2026-11-17
• expect no *.code.loca.zone entry; a wildcard would need DNS-01 validation with DNS-provider credentials that are not configured here, see reference/limits
• the vhost reads /etc/letsencrypt/live/code.loca.zone/fullchain.pem and privkey.pem, so a renewal that creates a second lineage such as code.loca.zone-0001 silently leaves nginx on the old files

7️⃣ Wiki host serves the built vault

cd /home/loca/dev/wikis && ./build.sh code
curl -sSI https://wiki.code.loca.zone/

• the build publishes a dated dist-* directory and repoints /home/loca/dev/wikis/code/current at it; that symlink is the nginx root for the wiki vhost
• expect HTTP/1.1 200 OK with no auth redirect once the build has published current; before the first build there is no current symlink, so nginx cannot serve the site
• ./build.sh code --check-only validates without publishing, which is the safe way to test frontmatter changes
• the wiki is public by design: Authelia default_policy is bypass and no rule names wiki.code.loca.zone

8️⃣ Initial user exists

curl -so /dev/null -w '%{http_code}\n' http://127.0.0.1:52226/api/v2/users/first

• expect 200: the initial Owner exists, which is upstream's own existence check. Observed on 2026-08-20, after the bootstrap flipped this endpoint from 404 to 200
• 404 is correct only on a fresh deployment that has never been bootstrapped; seeing it here means the postgres volume was lost and every user, template and workspace went with it
• no credential is ever recorded in this wiki; the account and the handling of its generated password are described in operations/auth-model

9️⃣ Template pushed and workspace running

coder login https://code.loca.zone && coder templates list && coder list
sudo -n docker ps --filter name=coder-loca-ws1 --format '{{.Names}} {{.Status}}'

• expect docker in the template list with an active version, and workspace ws1 owned by loca with status Started and health healthy; observed 2026-08-20
• expect exactly one workspace container coder-loca-ws1 in state Up; it is a sibling container on the host daemon rather than part of this compose project, so docker compose ps will never list it
• there is no coder binary on the host: the CLI lives inside the server container at /opt/coder, and the session file staged there during the bootstrap was deleted on purpose, so an in-container invocation needs a fresh login; the clean route is coder login from the operator's own machine
• a workspace stuck in Starting for several minutes is usually the image pull; the first build pulled codercom/enterprise-base:ubuntu in about 1m30s and started in about 95s in total, see operations/workspaces

🔟 Agent connected, app healthy, app URL loads

coder show ws1

• expect agent main connected with lifecycle ready, and app code-server with health healthy and subdomain false
• then open https://code.loca.zone/@loca/ws1.main/apps/code-server/ in a browser: expect a full code-server workbench, redirected to ?folder=/home/coder. Verified in a real browser on 2026-08-20, and this is the proof that path-based apps work with no wildcard access URL and no extra certificate
• the trailing slash matters; without it the server answers with a redirect
• a 502 or a workbench that never finishes connecting points at the agent rather than nginx; re-run coder show ws1 and read the agent logs inside the workspace container

1️⃣1️⃣ Resource pressure gate

free -m

• measured on 2026-08-19 before Coder existed: 36447 of 60278 MB used, and swap 15926 of 16383 MB consumed, alongside roughly ten co-tenant stacks including zulip, mattermost and eight postgres instances
• re-measured on 2026-08-20 immediately before the first workspace build: swap had drained to 0 of 16383 MB and roughly 31 GB of RAM was available, so the eight-gigabyte floor below passed on measurement rather than assumption; record free -m before and after every later build so the delta of one workspace stays known
• rule: do not create additional workspaces if available memory drops below about 8 GB
• container ceilings already in place: mem_limit: 8g on coder, mem_limit: 2g on database; see reference/limits

1️⃣2️⃣ Ad-hoc port hygiene

ss -ltn

• expect 127.0.0.1:52226 present and no other new listener from this stack
• any dev server started from a workspace or an IDE terminal must avoid every port in the AGENTS.md registry; 52222 pwiz, 52223 reserved for pwiz, 52224 cdsr and 52226 code are taken
• loca.zone application backends bind loopback inside 51000-53000

1️⃣3️⃣ Autostop TTL is armed

TOKEN=$(sed -n 's/^session_token=//p' /home/loca/dev/coder/.admin-credentials)
curl -s -H "Coder-Session-Token: $TOKEN" http://127.0.0.1:52226/api/v2/templates \
  | python3 -c "import json,sys; [print(t['name'], t['default_ttl_ms'], t['activity_bump_ms']) for t in json.load(sys.stdin)]"
curl -s -H "Coder-Session-Token: $TOKEN" http://127.0.0.1:52226/api/v2/users/loca/workspace/ws1 \
  | python3 -c "import json,sys; w=json.load(sys.stdin); print(w['ttl_ms'], w['latest_build']['deadline'])"

• expect template docker at 28800000 default TTL (8h) and 3600000 activity bump (1h)
• expect ws1 ttl_ms 28800000 and a non-null deadline roughly 8h ahead
• a deadline of None on a running workspace means that build predates the TTL: the schedule only attaches at build time, so coder restart ws1 -y is required once. coder schedule extend refuses with Workspace shutdown is manual. in that state
• the subcommand is coder schedule stop <ws> 8h; there is no override-stop in v2.35.3

1️⃣4️⃣ Nightly backup timer

systemctl list-timers stacks-backup --no-pager
sudo -n systemctl start stacks-backup.service
D=/home/loca/backups/stacks/$(date +%F)
sudo -n gunzip -t $D/coder-db.sql.gz && sudo -n ls -l $D

• expect the timer active with a next elapse at 04:30 local, Persistent=true so a missed run catches up after downtime
• expect three non-empty artifacts in the dated directory: coder-db.sql.gz (pg_dump of the Coder control plane), continue-config.tgz (/home/loca/.continue, which lives outside both projects), stack-configs.tgz (both stacks' config, deploy, scripts and templates)
• artifacts are root-owned; retention keeps 14 dated directories
• .env, .admin-credentials and .ai-credentials are deliberately excluded — they are keys, not data; verify with sudo -n tar tzf $D/stack-configs.tgz | grep -c credentials returning 0

1️⃣5️⃣ omp harness reaches Coder over MCP

printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"1"}}}' \
  | /home/loca/dev/coder/scripts/coder-mcp.sh | head -1
ls /tmp/tmp.* 2>/dev/null | wc -l
cd /home/loca/dev/coder && omp -p 'Call coder_get_authenticated_user and coder_list_workspaces. Report the username and each workspace name plus status.'

• expect the first line to contain "serverInfo":{"name":"Coder Agent","version":"v2.35.3+..."}
• expect the temp-file count to be unchanged, proving the wrapper's EXIT trap shredded the mode-600 env file holding the token
• expect the harness to answer from tools, not guesses: username loca, workspace ws1 running
• if the harness reports no tools, check ~/.omp/logs/omp.<date>.<pid>.log for MCP tool load failed ... Transport closed, which means the wrapper backgrounded the server and its stdin was redirected from /dev/null
• token expiry is the most likely 401 cause: the omp-mcp token lasts 168h maximum. Rotate with scripts/coder-token-rotate.sh and confirm with sudo -n docker exec coder /opt/coder tokens list
• 26 of 32 tools are exposed by design; template mutation is excluded. Confirm with a tools/list request and check that no coder_create_template*, coder_update_template_active_version, coder_delete_template, coder_delete_task or coder_upload_tar_file appears

🧹 Pre-publish wiki lint gate

Run this before every build.sh invocation, so before item 7 above. It is a gate, not a smoke test: it exits non-zero and publishes nothing.

python3 /home/loca/dev/wikis/lint-vault.py /home/loca/dev/coder/q5vault

• expect 11 pages, 0 findings, all 5 checks clean and exit code 0; --quiet suppresses that line so only failures print, which is the form to use in a wrapper
• the vault may also be named instead of pathed: python3 /home/loca/dev/wikis/lint-vault.py code resolves /home/loca/dev/wikis/code/content, the symlink into this vault
• why it exists: a frontmatter description: whose value held an unquoted colon-space made the Quartz note-properties transformer fail its YAML parse, and esbuild then died with a Go all goroutines are asleep - deadlock! panic rather than a readable error. One unquoted colon cost a whole build with no usable diagnostic
• it complements scripts/validate-ssot.py, which build.sh runs itself: that validator only gates wikis shipping a wiki-ia.json manifest, and this wiki ships none, so its SSOT Validation passed line is a vacuous pass here and this linter is the only real pre-publish gate on the vault
• check 1, frontmatter scalars: unquoted values containing a colon-space, starting with #, @, *, &, !, % or a backtick, ending in a colon, carrying an inline # comment that would silently truncate the value, or holding an unterminated quote. Reported as file:line plus the offending key
• check 2, wikilinks in tables: an unescaped | inside [[...]] on a table row splits the cell, and an alias containing / is truncated at the slash when rendered
• check 3, dead wikilinks: every target must resolve to a real .md in the vault, tried vault-root-relative first and then relative to the linking page, tolerating a leading ./ and an optional #anchor
• check 4, house style: no markdown bold anywhere
• check 5, page shape: no second H1 in one file, and no page without a frontmatter block
• checks 2 to 5 ignore fenced blocks and inline code spans, because nothing in those regions renders as markdown; a wikilink shown as an example is not a link
• standard library only, so it runs on any host with python3 and needs no pyyaml

📚 Sources

• https://coder.com/docs/install/docker
• https://coder.com/docs/admin/setup
• https://coder.com/docs/reference/api/users
• https://coder.com/docs/admin/networking/wildcard-access-url