coder/q5vault/operations/verification.md

---
title: "✅ Verification"
description: "Numbered proof checklist for the Coder stack with exact commands and expected output, including the deliberate contrast against the Authelia-gated cdsr.loca.zone host."
source: "verified deployment on the loca operator host, 2026-08-19; user, template, workspace and app checks verified 2026-08-20"
reviewed: 2026-08-20
---

# ✅ Verification

Run this list in order after any change to the compose file, the nginx vhost, the certificate or the image pin. Items 1 to 6 were observed on 2026-08-19 with the outputs shown; items 8, 9 and 10 were observed on 2026-08-20 after the first-user bootstrap, the template push and the first workspace build, and item 11 was re-measured the same day. Item 7 becomes true again on every wiki build.

## 1️⃣ Both containers up, database healthy

```bash
cd /home/loca/dev/coder && sudo -n docker compose ps
```

- expect container `coder` with state `running` and the single port mapping `127.0.0.1:52226->7080/tcp`
- expect container `coder-database` with state `running` and health `healthy`, and no port mapping at all
- a `coder-database` stuck in `starting` means the `pg_isready -U coder -d coder` healthcheck has not passed yet; `coder` will not start until it does

## 2️⃣ Server health on loopback

```bash
curl -sf http://127.0.0.1:52226/healthz
```

- expect exactly `OK`
- a non-zero exit from `curl -sf` means the server is not answering; check `sudo -n docker compose logs --tail 200 coder`

## 3️⃣ Running build matches the pin

```bash
curl -sf http://127.0.0.1:52226/api/v2/buildinfo
```

- expect `version` `v2.35.3+65e2bfb`, matching `config/runtime-version.txt` (`v2.35.3`) and the image tag in `docker-compose.yml`
- expect `dashboard_url` `https://code.loca.zone`, which proves `CODER_ACCESS_URL` reached the server
- expect `telemetry` `false`, which proves `CODER_TELEMETRY_ENABLE` took effect
- no jq on this host; if the raw JSON needs shaping, use `python3 -c` with the `json` module

## 4️⃣ Public app host answers directly, with no auth redirect

```bash
curl -sSI https://code.loca.zone/
```

- expect `HTTP/1.1 200 OK` (or `HTTP/2 200`), served by the Coder SPA, with no `Location` header
- expect the security headers from `snippets/security-headers-app.conf`
- a `302` to `https://auth.loca.zone/...` here is a regression: someone added an Authelia rule for `code.loca.zone`. Remove it, reload Authelia, and re-read [[operations/auth-model]]

## 5️⃣ Contrast check against the gated sibling host

```bash
curl -sSI https://cdsr.loca.zone/
```

- expect `HTTP/1.1 302` with a `Location` pointing at `https://auth.loca.zone/`, because `cdsr.loca.zone` carries an explicit `one_factor` Authelia rule
- this is the control for item 4: the same nginx and the same Authelia produce a redirect there and a direct `200` here, which proves the difference is the per-host rule and not a broken gate
- if both hosts return `200`, Authelia itself is down or its rule set was truncated; if both return `302`, the Coder vhost was gated by mistake

## 6️⃣ Certificate lineage covers both names

```bash
sudo -n certbot certificates
```

- expect one lineage named `code.loca.zone` whose domains are `code.loca.zone` and `wiki.code.loca.zone`, expiring 2026-11-17
- expect no `*.code.loca.zone` entry; a wildcard would need DNS-01 validation with DNS-provider credentials that are not configured here, see [[reference/limits]]
- the vhost reads `/etc/letsencrypt/live/code.loca.zone/fullchain.pem` and `privkey.pem`, so a renewal that creates a second lineage such as `code.loca.zone-0001` silently leaves nginx on the old files

## 7️⃣ Wiki host serves the built vault

```bash
cd /home/loca/dev/wikis && ./build.sh code
curl -sSI https://wiki.code.loca.zone/
```

- the build publishes a dated `dist-*` directory and repoints `/home/loca/dev/wikis/code/current` at it; that symlink is the nginx root for the wiki vhost
- expect `HTTP/1.1 200 OK` with no auth redirect once the build has published `current`; before the first build there is no `current` symlink, so nginx cannot serve the site
- `./build.sh code --check-only` validates without publishing, which is the safe way to test frontmatter changes
- the wiki is public by design: Authelia `default_policy` is `bypass` and no rule names `wiki.code.loca.zone`

## 8️⃣ Initial user exists

```bash
curl -so /dev/null -w '%{http_code}\n' http://127.0.0.1:52226/api/v2/users/first
```

- expect `200`: the initial Owner exists, which is upstream's own existence check. Observed on 2026-08-20, after the bootstrap flipped this endpoint from `404` to `200`
- `404` is correct only on a fresh deployment that has never been bootstrapped; seeing it here means the postgres volume was lost and every user, template and workspace went with it
- no credential is ever recorded in this wiki; the account and the handling of its generated password are described in [[operations/auth-model]]

## 9️⃣ Template pushed and workspace running

```bash
coder login https://code.loca.zone && coder templates list && coder list
sudo -n docker ps --filter name=coder-loca-ws1 --format '{{.Names}} {{.Status}}'
```

- expect `docker` in the template list with an active version, and workspace `ws1` owned by `loca` with status `Started` and health `healthy`; observed 2026-08-20
- expect exactly one workspace container `coder-loca-ws1` in state `Up`; it is a sibling container on the host daemon rather than part of this compose project, so `docker compose ps` will never list it
- there is no `coder` binary on the host: the CLI lives inside the server container at `/opt/coder`, and the session file staged there during the bootstrap was deleted on purpose, so an in-container invocation needs a fresh login; the clean route is `coder login` from the operator's own machine
- a workspace stuck in `Starting` for several minutes is usually the image pull; the first build pulled `codercom/enterprise-base:ubuntu` in about 1m30s and started in about 95s in total, see [[operations/workspaces]]

## 🔟 Agent connected, app healthy, app URL loads

```bash
coder show ws1
```

- expect agent `main` connected with lifecycle `ready`, and app `code-server` with health `healthy` and `subdomain` false
- then open https://code.loca.zone/@loca/ws1.main/apps/code-server/ in a browser: expect a full code-server workbench, redirected to `?folder=/home/coder`. Verified in a real browser on 2026-08-20, and this is the proof that path-based apps work with no wildcard access URL and no extra certificate
- the trailing slash matters; without it the server answers with a redirect
- a `502` or a workbench that never finishes connecting points at the agent rather than nginx; re-run `coder show ws1` and read the agent logs inside the workspace container

## 1️⃣1️⃣ Resource pressure gate

```bash
free -m
```

- measured on 2026-08-19 before Coder existed: 36447 of 60278 MB used, and swap 15926 of 16383 MB consumed, alongside roughly ten co-tenant stacks including zulip, mattermost and eight postgres instances
- re-measured on 2026-08-20 immediately before the first workspace build: swap had drained to 0 of 16383 MB and roughly 31 GB of RAM was available, so the eight-gigabyte floor below passed on measurement rather than assumption; record `free -m` before and after every later build so the delta of one workspace stays known
- rule: do not create additional workspaces if available memory drops below about 8 GB
- container ceilings already in place: `mem_limit: 8g` on `coder`, `mem_limit: 2g` on `database`; see [[reference/limits]]

## 1️⃣2️⃣ Ad-hoc port hygiene

```bash
ss -ltn
```

- expect `127.0.0.1:52226` present and no other new listener from this stack
- any dev server started from a workspace or an IDE terminal must avoid every port in the AGENTS.md registry; `52222` pwiz, `52223` reserved for pwiz, `52224` cdsr and `52226` code are taken
- loca.zone application backends bind loopback inside `51000-53000`

## 1️⃣3️⃣ Autostop TTL is armed

```bash
TOKEN=$(sed -n 's/^session_token=//p' /home/loca/dev/coder/.admin-credentials)
curl -s -H "Coder-Session-Token: $TOKEN" http://127.0.0.1:52226/api/v2/templates \
  | python3 -c "import json,sys; [print(t['name'], t['default_ttl_ms'], t['activity_bump_ms']) for t in json.load(sys.stdin)]"
curl -s -H "Coder-Session-Token: $TOKEN" http://127.0.0.1:52226/api/v2/users/loca/workspace/ws1 \
  | python3 -c "import json,sys; w=json.load(sys.stdin); print(w['ttl_ms'], w['latest_build']['deadline'])"
```

- expect template `docker` at `28800000` default TTL (8h) and `3600000` activity bump (1h)
- expect `ws1` `ttl_ms` `28800000` and a non-null `deadline` roughly 8h ahead
- a `deadline` of `None` on a running workspace means that build predates the TTL: the schedule only attaches at build time, so `coder restart ws1 -y` is required once. `coder schedule extend` refuses with `Workspace shutdown is manual.` in that state
- the subcommand is `coder schedule stop <ws> 8h`; there is no `override-stop` in v2.35.3

## 1️⃣4️⃣ Nightly backup timer

```bash
systemctl list-timers stacks-backup --no-pager
sudo -n systemctl start stacks-backup.service
D=/home/loca/backups/stacks/$(date +%F)
sudo -n gunzip -t $D/coder-db.sql.gz && sudo -n ls -l $D
```

- expect the timer `active` with a next elapse at 04:30 local, `Persistent=true` so a missed run catches up after downtime
- expect three non-empty artifacts in the dated directory: `coder-db.sql.gz` (`pg_dump` of the Coder control plane), `continue-config.tgz` (`/home/loca/.continue`, which lives outside both projects), `stack-configs.tgz` (both stacks' config, deploy, scripts and templates)
- artifacts are root-owned; retention keeps 14 dated directories
- `.env`, `.admin-credentials` and `.ai-credentials` are deliberately excluded — they are keys, not data; verify with `sudo -n tar tzf $D/stack-configs.tgz | grep -c credentials` returning `0`

## 1️⃣5️⃣ omp harness reaches Coder over MCP

```bash
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"probe","version":"1"}}}' \
  | /home/loca/dev/coder/scripts/coder-mcp.sh | head -1
ls /tmp/tmp.* 2>/dev/null | wc -l
cd /home/loca/dev/coder && omp -p 'Call coder_get_authenticated_user and coder_list_workspaces. Report the username and each workspace name plus status.'
```

- expect the first line to contain `"serverInfo":{"name":"Coder Agent","version":"v2.35.3+..."}`
- expect the temp-file count to be unchanged, proving the wrapper's EXIT trap shredded the mode-600 env file holding the token
- expect the harness to answer from tools, not guesses: username `loca`, workspace `ws1` `running`
- if the harness reports no tools, check `~/.omp/logs/omp.<date>.<pid>.log` for `MCP tool load failed ... Transport closed`, which means the wrapper backgrounded the server and its stdin was redirected from `/dev/null`
- token expiry is the most likely 401 cause: the `omp-mcp` token lasts 168h maximum. Rotate with `scripts/coder-token-rotate.sh` and confirm with `sudo -n docker exec coder /opt/coder tokens list`
- 26 of 32 tools are exposed by design; template mutation is excluded. Confirm with a `tools/list` request and check that no `coder_create_template*`, `coder_update_template_active_version`, `coder_delete_template`, `coder_delete_task` or `coder_upload_tar_file` appears

## 🧹 Pre-publish wiki lint gate

Run this before every `build.sh` invocation, so before item 7 above. It is a gate, not a smoke test: it exits non-zero and publishes nothing.

```bash
python3 /home/loca/dev/wikis/lint-vault.py /home/loca/dev/coder/q5vault
```

- expect `11 pages, 0 findings, all 5 checks clean` and exit code `0`; `--quiet` suppresses that line so only failures print, which is the form to use in a wrapper
- the vault may also be named instead of pathed: `python3 /home/loca/dev/wikis/lint-vault.py code` resolves `/home/loca/dev/wikis/code/content`, the symlink into this vault
- why it exists: a frontmatter `description:` whose value held an unquoted colon-space made the Quartz note-properties transformer fail its YAML parse, and esbuild then died with a Go `all goroutines are asleep - deadlock!` panic rather than a readable error. One unquoted colon cost a whole build with no usable diagnostic
- it complements `scripts/validate-ssot.py`, which `build.sh` runs itself: that validator only gates wikis shipping a `wiki-ia.json` manifest, and this wiki ships none, so its `SSOT Validation passed` line is a vacuous pass here and this linter is the only real pre-publish gate on the vault
- check 1, frontmatter scalars: unquoted values containing a colon-space, starting with `#`, `@`, `*`, `&`, `!`, `%` or a backtick, ending in a colon, carrying an inline ` #` comment that would silently truncate the value, or holding an unterminated quote. Reported as `file:line` plus the offending key
- check 2, wikilinks in tables: an unescaped `|` inside `[[...]]` on a table row splits the cell, and an alias containing `/` is truncated at the slash when rendered
- check 3, dead wikilinks: every target must resolve to a real `.md` in the vault, tried vault-root-relative first and then relative to the linking page, tolerating a leading `./` and an optional `#anchor`
- check 4, house style: no markdown bold anywhere
- check 5, page shape: no second H1 in one file, and no page without a frontmatter block
- checks 2 to 5 ignore fenced blocks and inline code spans, because nothing in those regions renders as markdown; a wikilink shown as an example is not a link
- standard library only, so it runs on any host with `python3` and needs no pyyaml

## 📚 Sources

- https://coder.com/docs/install/docker
- https://coder.com/docs/admin/setup
- https://coder.com/docs/reference/api/users
- https://coder.com/docs/admin/networking/wildcard-access-url