Skip to main content

INVESTIGATE backlog — priority view

Purpose: triage tool, not a roadmap. Decides what to investigate next — not what to build next. The 38 INVESTIGATE files in backlog/ were written at different times for different reasons; this doc separates the ones ready to be done from the ones that should wait, and orders the ready ones by what they unblock.

Last updated: 2026-09-18 (fifteenth refresh). Re-rank whenever an INVESTIGATE moves to completed/, a child PLAN ships, or a new INVESTIGATE lands.

How to read the tiers: tier order is the order to start the investigation, not the order to finish. Tier 1 means "next on deck"; Tier 4 means "don't open this yet — wait for prereqs or product clarity." Tier 0 is "in flight — no fresh investigation work needed but the file still lives here because work isn't fully shipped."

UIS lifecycle convention: an INVESTIGATE moves from backlog/ to completed/ once every child PLAN has shipped (or the investigation is otherwise closed). Once moved, it disappears from this doc — see completed/ for the historical list.

Decision — how applications get deployed (Terje, 2026-09-07)​

Answered in writing for the first time, and it settles the question that urb-agents#163 was opened to ask. Recorded normatively in Rules for Deploying Applications.

uis provisions. ArgoCD deploys. The seam is uis configure, which writes a Secret the application's own manifest reads.

Both mechanisms are permanent. The choice is per-object, not per-application — one application can need both, and the first real one does: Atlas's pipeline has no workload of its own (Dagster launches its image), while its frontend is an ordinary Deployment and belongs to ArgoCD.

No UIS Application type. Deferred per ANALYSIS-nais-uis §4 item 13 — and it would not have covered Atlas anyway: NAIS's two workload kinds both run the application's own pods, and there is no NAIS concept for handing an image to a shared orchestrator. Atlas's shape is a consequence of UIS making Dagster a multi-tenant platform service, which NAIS has no analogue for.

The sequence, and the dependency people get wrong​

WorkEffortState
1Atlas via templates — templates-000 ordering, then the config: fieldsMin flight
2Atlas frontend via uis argocd registerfreeready, blocked by nothing
3Retract SCRIPT_CONFIGURABLE where no handler existsSfiled today
4Per-workload named secrets (§4 item 1)Mnot started
5Reconsider a UIS Application typeLdeferred

⚠️ "We need GitOps before we onboard developers" is the wrong dependency. ArgoCD already exists and already self-heals. What a developer cannot get today is a database with a credential in a declaration — that is item 4, and it gates external developers and whole-lifecycle GitOps alike. 🔴 And the limit to state honestly meanwhile: because uis configure mints a password UIS deliberately does not store, a cluster rebuilt from git alone comes up with no application secrets.

Item 3 is the cheapest thing here and the one that was not on anyone's list. Eight services advertise configure; two have handlers. The two that work are exactly the two Atlas needs — so the provisioning half of the rule works for the first tenant and fails for the second. Retract the six; build none on speculation.

Decision — Traefik v3 is a requirement (Terje, 2026-09-18)​

Stated for the first time, and it was already true. Every Traefik object in manifests/ uses apiVersion: traefik.io/v1alpha1 — the v3 CRD group — and none uses v2's traefik.containo.us (34 of 34, counted). So UIS has required v3 since the manifests were written; what was missing was saying so and checking it.

Where the two platforms stand:

PlatformTraefikHow
rancher-desktopv3.6.13k3s manages it via a HelmChart CR; 003-setup-traefik.yml detects that and skips the helm install
AKS / GCP / AWS / bare k8schart 39.0.7UIS installs it, pinned for rancher-desktop parity (traefik-39.0.701+up39.0.7, proxy 3.6.13)

🔴 The gap is that nothing asserts the running version. On rancher-desktop UIS installs nothing, so the version is whatever k3s shipped — and applying v3 CRDs to a v2 controller produces the failure mode recorded in #199: every service 404s while the playbooks report success and exit 0. Same shape as the three Cloudflare commands fixed in 1.6.118 — the system has the information and reports the opposite.

⚠️ The UIS-managed install is the half that needs work (Terje: "on other systems we must install traefik into uis. here we need to do an update"). The pin exists for parity, so it is only correct as long as it tracks what rancher-desktop ships; the playbook's own note says the two "migrate together as one coordinated change" when k3s moves to chart v40. Nothing enforces that either — it is a comment, not a check.

What the decision unblocks: PathRegexp is v3-only, and per-path auth policies need it — one gate for a host and a stricter gate for a subset of its paths. Without v3 that has to be expressed as a path prefix or a separate hostname, which would mean changing a tenant's existing URL convention. With v3 stated, the tenant's convention survives untouched. See PLAN-service-oauth2-proxy.

Current status — tor-agent, 2026-09-10​

🟢 uis template install atlas works end to end on a machine that has never seen it. Terje's assertion — install, then materialise, with no human step anywhere — was met on 2026-09-10 (imac, urb-agents#495): absence proved, EXIT=0, raw/_migrations RUN_SUCCESS, and every figure back to the pre-teardown baseline (47 raw tables, 60 marts, 13 views, 122 rows). Installed from the published catalogue, not a staged file.

1.6.24 → 1.6.51, almost all of them defects found by imac grading UIS as a novice from a factory reset. The pattern worth recording: the platform stopped being the blocker at 1.6.35, and everything since has been the difference between works and usable by someone who has never seen it.

🔴 Nine things are open and none of them are atlas.

system-topology-coverage — outcome 2, exercise the proxy topologyTier 1, oldest live requirement. ⚠️ This was invisible in my published status until ops-dev reconciled a dying ledger (urb-agents#544) and could not find it. The artefact was always here; the summary was thinner than the tracking behind it
Container-image scanning, 46 imagesunstarted, a platform decision. ⚠️ Requirement to keep: "no findings" must render differently from "the scan did not run"
cli-load-and-report-on-application-datauis dagster materialize, uis template status, and the uis status URL column. Filed from imac's novice grading; nothing blocks it but my ordering
cli-postgresql-target-tenancyNEW 2026-09-10. UIS hardcodes its Postgres host and connects as superuser, so it cannot tell a database it owns from a shared server it is a guest on. Harmless today because nothing drops a database — which means configure-postgresql-purge is the plan that arms it. Filed before that work rather than during it, on purpose
cli-init-file-partial-applyNEW 2026-09-10. A failing init: on an existing database is documented and commented as leaving it alone; psql runs without --single-transaction, so everything before the failure is committed. The wording is corrected in 1.6.48; whether to make the sentence true with one flag is a real trade (CREATE INDEX CONCURRENTLY cannot run in a transaction) and is the plan
system-public-repo-internal-detailNEW 2026-09-10. This repo is public and the shipping surface named a lab address, container id and bridge. Scrubbed and linted in 1.6.50; what remains is 🔴 Terje's call — whether hosts/<name>/ and the historical plan records belong in a public repository at all
cli-unmarked-proxy-silent-overlayNEW 2026-09-10. A pre-marker proxy with no StatefulSet is the one revert path that warns nothing: the in-cluster workload deploys beside a live relay and both back the Service. Found tracing #600's blast radius; the gate above it is fixed, this branch is not
docs-website-dependency-alertsNEW 2026-09-10. Four open alerts on the docs site, 2 high. ⚠️ Upgrading Docusaurus does not clear them — image-size has no patched version and 3.10.2 still requires it, so npm audit's sixteen proposed fixes are noise. qs is cheap and real; image-size is upstream's
system-ci-self-hosted-runner-readinessNEW 2026-09-10. Terje asked for a self-hosted runner because Actions credits are exhausted. ⚠️ Measured: this repo is PUBLIC and was building containers an hour after the private repo went dark — public repos are not billed, so the emergency is confined to the private ones. Three properties would have to be fixed before any migration, one of them a CI job that pushes to main

What the last three days actually taught, beyond the fixes​

Two guards now make failure audible, and they have each caught something care had already missed:

  • the test framework reconciles skips against starts and fails a suite that cannot account for its own tests — it caught a lint of mine that resolved a path one directory short, found nothing, and reported ALL TESTS PASSED
  • lints that fail on a shape rather than an instance — the kubectl wait one, the bashism one, the set -e one that was widened after it missed the spelling it existed for

imac's framing, which is the reason to prefer them over care: "you did not catch it by being careful, you caught it because you had built something that makes silence visible."

And three rules from other agents, each sharper than my own version:

  • a guard that checks a field is well-formed does not check that it is true (atlas) — the general form of what my structural tests kept getting wrong
  • a check that cannot pass is a different defect with the opposite sign from one that cannot fail (imac) — failing closed costs one confusing exit code; failing open cost four gradings
  • when a measurement answers a yes/no question, the finding is usually in the mechanism, not the answer — exit 1 on no matching resources answered "no vacuous pass" and also said "fails before the resource exists", which broke first installs for two versions

Previous status — tor-agent, 2026-09-07​

state: one investigation filed, unstarted. active/ is still empty.

The first real application arrived, and it changed what the top of this queue is for. The atlas agent asked, through urb-agents#159, whether UIS should grow an Application type to install an app that needs a database, a Dagster code location and a PostgREST instance as one thing. The answer was no — and the reason is that this repository had already priced that decision: ANALYSIS-nais-uis §4 ranks a UIS Application manifest last of thirteen, at L effort, explicitly deferred behind three capabilities that have not landed.

⚠️ Two of the four "missing" steps in that request already existed and had never been used. ./uis configure postgresql --namespace <ns> --secret-name-prefix <p> creates the per-app database, its owning role, applies migrations from stdin with rollback, and writes the secret into another service's namespace. The first application it was built for did not use it, because docs/services/analytics/dagster.md told tenants "UIS does not create them for you."

That is the finding of the day, and it is not really about Dagster: a capability that is documented not to exist does not exist. The doc defect is fixed; the class it belongs to is worth watching, because SCRIPT_CONFIGURABLE advertises the reverse shape — six services claim a handler that is missing (ANALYSIS-nais-uis §4 item 4).

What came out of it​

What
templates: multi-surface applicationNEW, Tier 1. uis template install is already the unit above a service — provides:, params:, deploy-then-configure in priority order. Four specific things stop it installing this tenant. One of them, TPL-F3, is a latent defect for any multi-instance service: template.sh:405 deploys without --app while the configure call two lines later passes it
docs: the two secret pathsFixed in the same change. uis configure yields key DATABASE_URL in a secret named <prefix>-db; kubectl yields whatever you choose. Both are valid, nothing said they diverge, and the divergence is invisible until a run pod fails to connect

Not taken on, deliberately​

Reaching an installation from a second machine on the same LAN is not a UIS feature. Traefik already routes every app on HostRegexp(`<prefix>\..+`), which matches any suffix, and PostgREST already sets PGRST_SERVER_CORS_ALLOWED_ORIGINS: "*" — so it needs a Host header and a name that resolves, not platform work. Same ruling as roaming-dependency-addresses: installation implementation, not UIS. The one unmeasured thing is whether a given host's Traefik listens off loopback; if it does not, that is a new investigation and not a fold-in.

⚠️ The proxy plan reopened after it closed, and that is the finding of the day. It was closed against a laptop fixture. The platform manager then ran the new verify against the production installation, read-only, and it failed while production was working perfectly — a proxy created before the marker existed. Four more defects followed, three more tester rounds, all merged. Phase 4 of the plan records them.

Every one was reachable only from an installation that predates the feature. That is exactly the gap the topology-coverage requirement was written about, and it is the second time this week that production found what no fixture could.

Two workstreams closed 2026-08-30​

1. The external-services proxy — six defects across three rounds, all merged. The proxy now takes over the Service it stands in for, the verify proves which topology answered rather than reading it from the config file, and the round trip returns cleanly. Recorded in PLAN-system-external-services-proxy-takeover.

2. The Node 20 runtime bump — 19 pins across four workflows, graded 8/8. Recorded in PLAN-system-actions-v5-runtime-bump. It also produced the 1.6.5 release, because the bump republished the container with a moved digest and an unchanged version.txt, which ./uis pull could not see.

Three investigations filed 2026-08-30, none started​

Each was a class found underneath a specific fix, filed rather than patched away:

What
workflow paths-filter driftFour instances of a paths: filter narrower than what the job depends on. Three found by the tester, the fourth by an audit that took under a second. The failure is silence — nothing goes red, work simply is not done
launcher/image version driftNow carries a live instance: an image republished with a moved digest and an unchanged version, invisible to the update mechanism. Adds the open question of whether the version bump should be automatic
deploy revert exit codeuis deploy reports one exit code for two separable outcomes; the same revert gave exit 0 and exit 3 minutes apart

Next, in order — the ordering is mine​

  1. Vulnerability scanning for the 46 running images. The last requirement in the original queue that has not been started. The code half is already tracked at a documented floor, so only the pods half is open.
  2. A CI guard for the disclosure boundary — a publicly routable address or a credential-shaped string should fail the build. Today that boundary is held by review alone, and Terje's decision to accept the existing internal addresses rests on it.
  3. The decidable half of the paths-filter class — every workflow must list its own file. Cheap, mechanically checkable, and the tester independently proposed the same property.

⚠️ Where testing happens — Terje, 2026-09-05​

Testing happens on the tester's cluster. Not on the reference installation. If work here needs a cluster to fix or verify, it is asked of the tester, and that is now the rule rather than a workaround.

This retires a caveat this file carried seven times. Every closure above ended with some form of "the reference installation runs the external-services proxy shape and nothing on it has been exercised" — written as a gap still to close. It is no longer a gap. It is a decision about where the fleet tests, and the honest statement is narrower:

  • What the work is verified against: the tester's cluster, which is version-matched to production and which built the proxy fixture that found six of the eight proxy defects.
  • What remains true and is no longer a deficiency: the reference installation has not been exercised by this work. That is now by design, so it should not be read as an outstanding item, and I will stop appending it to closures as though it were one.
  • What would change this: a defect that only the reference installation's shape can produce. One such defect already happened — the pre-marker proxy, found by a read-only look at production that no fixture could reproduce. So "we test on the tester's cluster" is a decision about cost, not a claim that the two are equivalent, and that distinction is worth keeping.

What changed in this refresh (2026-05-07 → 2026-08-11)​

The doc had gone three months without a re-rank, against a rule that says re-rank whenever a plan ships. In that window 12 investigations landed and roughly 9 plans shipped, so the triage view had drifted badly from the repo:

  • 12 investigations were missing entirely, including the two driving all current work — system-observability and service-uptime-kuma. The doc claimed 25 files; there were 35.
  • One row pointed at a file that had already shipped: cli-undeploy-purge-flag now lives in completed/. Row deleted per rule 4.
  • system-verification-playbooks-usage is promoted from Tier 2 to Tier 1. It predicted, in March, the exact defect found on 2026-08-11: a verify playbook present in the repo and wired into nothing. See its Tier 1 entry.
  • Tier 0 grew from 3 rows to 7 — it was hiding how much is genuinely in flight.

What changed 2026-08-13​

  • Observability landed on main but is not finished — 55 tasks done, 56 open. See its Tier 0 row. The gate defect (OBS-F6) is closed and verified against the shipped image; the automation the platform promises is not.

  • Tier 1 #2 gained a third and much more serious finding. kubectl run --rm -i can return rc=0 with empty stdout. Any verify playbook asserting on stdout can therefore report a passing service as failing — silently, and load-dependently. This idiom is used well beyond the playbook where it was found.

  • The docs site can now be built locally after all, which changes a standing assumption. There is no Node on the workstation or in the provision-host container, so adding-a-service step 11 (npm run build) had been skipped every time it was asked for. It runs fine in a container:

    cd website && docker run --rm -u $(id -u):$(id -g) \
      -v "$PWD":/w -w /w -e HOME=/tmp -e CI=true node:20 \
      sh -c 'npm ci && npm run build'

    The guide should say this. A required step that cannot be run on the machines people actually use will keep being skipped, and it was.

  • Website vulnerabilities went 99 → 2 (26 Dependabot PRs, then npm overrides for two transitive roots Dependabot cannot reach). The remaining two are image-size with no published patch — a floor, not a backlog item.

  • The first local build surfaced pre-existing broken links and one broken anchor in the docs. Warnings, not failures. Currently owned by nobody.

What changed 2026-09-07 (twelfth refresh) — the first application tenant, and A over B​

One investigation filed, one documentation defect fixed, one architectural option refused.

Filed: templates-multi-surface-application, Tier 1. Six findings and five open questions about extending uis template install to cover an application that spans a database, a Dagster code location and a per-app PostgREST instance.

Refused: a UIS Application type, on this repository's own prior reasoning rather than on fresh judgement. That matters more than the refusal — ANALYSIS-nais-uis was written 2026-08-15 as an analysis with no approved work in it, and three weeks later it answered a live design question with a ranking, an effort estimate and a stated precondition. An analysis that prices deferred options earns its keep the first time someone asks for one of them.

Fixed: docs/services/analytics/dagster.md said "UIS does not create them for you" about secrets that uis configure has been able to create since PLAN-001. The two creation paths yield different environment-variable names (DATABASE_URL versus one you choose) and nothing said so.

The pattern worth carrying forward: the request named four missing capabilities and two of them already shipped. Before designing anything for a tenant, read what the CLI can already do — and distrust the docs in both directions, since SCRIPT_CONFIGURABLE advertises the opposite error (six services claim a handler that does not exist).

What changed 2026-08-30 (tor-agent, UIS maintainer)​

  • The broken-link item above is CLOSED. It was the last bullet of the 2026-08-13 refresh and had been nobody's for a fortnight. onBrokenLinks, onBrokenAnchors and onBrokenMarkdownLinks are now 'throw', the one broken anchor is fixed, and the build runs on pull requests — so a broken link fails the PR rather than reding main. Graded PASS by the independent tester in two rounds (9e79334, 4ebee23).
  • The container-build recipe above is no longer the only option. That bullet says there is no Node on the workstation or in the provision-host container. There is Node on the maintainer machine now, and cd website && npm run build runs natively in about 31 s warm. The container recipe still works and is still right for machines without Node; it is no longer the only route, and the pre-flight is documented in documentation.md.
  • A stray 1priority.md existed for two hours and has been deleted. ops's first status poll asked every agent to create plans/backlog/1priority.md; that was ops's error, and on a case-insensitive filesystem the lowercase twin would have silently overwritten this file. I wrote to plans/1priority.md instead so nothing was lost, and the revised poll confirmed there is no new file. Its content is folded into the status section at the top of this document, and the stray is removed.

What changed 2026-08-26 (sixth entry) — topology coverage gets a file​

  • system-topology-coverage filed. A requirement from ops (25/8, at Terje's direction) that had been accepted and "queued" for a day while existing only as a message in terchris/home. It had a queue position and no artefact — the same shape as the defects it is about, where an absent record reads as a live one. Outcomes 1 and 4 (target_host plumbing, loud configuration errors) already shipped via lib/verify-target.sh; outcome 2 is open — nothing yet exercises the external-postgres proxy topology.
  • Version system shipped (1.5.2 → 1.6.3): one version.txt, remote-version detection on every command, display in help/status and on the website navbar, launcher self-update, and untagged-image cleanup. Verified by ops on asgard. UIS stays on 1.6.x until every service runs on asgard — Terje, 25/8.
  • TALK protocol v2 adopted. Messages now live in ai-developer/talk/ in terchris/home and rounds append under ## Round N rather than overwriting.

What changed 2026-08-23 (fifth entry) — browserless shipped, Dagster shipped​

Both were built through the new build/verify-separated loop: assist builds, the imac tester verifies independently on a prod-matched cluster. Two very different results, and the difference is the point.

  • browserless-001 shipped — PASS on all four criteria, first attempt, zero defects returned. The AUTOMATION category is live; browserless is the first service in it. This is the fix for the gap noted in the 2026-08-21 (second entry) below: Uptime Kuma's real-browser monitor type was dead in every installation because no Chrome existed to connect to. It now has one.
  • dagster-001 shipped — but took three rounds and six defects. Row 27 in Tier 4 is updated accordingly.
  • The difference between one round and three was where the bugs were found. browserless had two defects of exactly the same class as Dagster's — but both were caught by the builder's own smoke checks before handover, because the build ran a real render rather than reading a health endpoint. Dagster's D2, D3 and D5 all reached the tester because the verify checked whether a command succeeded rather than whether the system reached the right state.
  • Concretely: browserless spent part of its build 1/1 Running, zero restarts, with /pressure reporting "isAvailable": true — and every render failing. A volume mount had shadowed the browser's data directory. No health signal moved. Its verify therefore renders a page and asserts a marker comes back, and that assertion was tested against the bug before handover: reintroducing the mount makes it fail while the pod stays healthy.
  • This is the third independent confirmation for Tier 1 #2 (verification-playbooks-usage), now arriving from a service that had no verify playbook at all until this week. The investigation predicted "deployments report success when no real validation happened"; a green /pressure on a browser that cannot render is precisely that.
  • New: uis-docs-writes-outside-repo. ./uis docs run from a workstation checkout writes to /mnt/urbalurbadisk/…, prints ✓ Generated … (34 services) and exits 0 while leaving the repo untouched — so a newly added service silently never reaches services.json. Found the hard way here. Same family as Tier 1 #2 and the test-all gap: a command reporting on its own execution rather than on the state it produced.
  • Still open: neko-001 is next and depends on the AUTOMATION category this created.

What changed 2026-08-21 (fourth entry) — validation run, three fixes, three plans​

Eighteen services validated on a factory-reset Rancher Desktop. Four bugs found and three fixed the same day; see STATUS-rancher-desktop-validation-2026-08-21.

  • Fixed and pushed: the Backstage catalog generator emitting invalid ConfigMap keys from a trailing comment (3aa3ef2); Backstage resolving a floating third-party plugin index, plus its chart pinned (c7dae01); and a deploy following an undeploy dying on the terminating namespace (91fd276).
  • New: helm-chart-version-pinning. Terje's call after Backstage broke on a floating dependency. Measured: 16 of 24 Helm-based services have no --version — two thirds of the observability stack among them. Backstage was the worst shape of all: image pinned, chart floating, so the pairing that ran was never the pairing that was tested. Phase 3 records the wider lesson — a pinned chart is not a pinned deploy, since Backstage's actual failure was an artefact the chart fetched at install time.
  • New: test-all-coverage-and-dry-run. --dry-run omits the clean phase, so it shows the constructive half of the plan and hides the destructive half. Plus gravitee skipped permanently with no date or issue, and SKIP_SERVICES_CONDITIONAL empty.
  • The recurring shape, now three-for-three: the failure names the wrong service. OBS-F6 blamed Grafana; the namespace race blamed enonic; Backstage went through two wrong causes. Each cost a wrong theory first, because the obvious next step — investigate the service named — is always wrong.
  • Counter-evidence for test-all: it found the namespace race, which nothing else would have. An argument for running it more, and therefore for it being honest about what it skips.

What changed 2026-08-21 (third entry) — approvals, sequencing, and two bugs filed​

  • Terje approved the two-service split, the AUTOMATION category, and neko as a planned service — with browserless shipping first. Both plans updated from "awaiting approval" to approved-and-sequenced.
  • Both plans were rewritten from scratch after Terje asked whether I had read adding-a-service.md and the rules docs. I had not, well enough — they were written from reading service definitions and inferring the rest. They missed the category registry (categories.sh), enabled-services.conf, stacks.sh and sidebars.ts; used -config.yaml for manifests that are not Helm values; numbered an IngressRoute against the wrong playbook; and did not know SCRIPT_PLAYBOOK/SCRIPT_MANIFEST are mutually exclusive. Recorded in the browserless plan's Implementation Notes rather than quietly fixed, because "wrote a plan from inference and it looked complete" is the failure worth remembering.
  • New: provisioning-unsafe-test-idiom. rules/provisioning.md presents the kubectl run --rm -i + until: on stdout idiom as the CORRECT pattern, in two sections. That is Tier 1 #2's third finding, and this is its propagation source — it explains why the idiom "appears across other services' verify playbooks." Recorded on the investigation too; the plan fixes the source and hands the occurrence list back rather than sweeping services itself.
  • verify-registration-fix gained task 2.4 — a fourth hand-maintained list, the usage text inside cmd_verify(). Currently in sync, so drift-prevention rather than a live defect; better derived from VERIFY_SERVICES than documented as a fourth place to edit.
  • A process note worth keeping: I cited PLAN-cli-verify-registration-fix by filename in two plans without opening it, then reported its contents back as a discovery. Search found the file; I did not read it. Citing is not reading.

What changed 2026-08-21 (second entry) — the browser platform becomes a service​

  • Terje decided the browser platform ships as a UIS service. Not reopened. What was open — name, category, scope — is proposed in two plans awaiting his approval: browserless-001 and neko-001. Nothing is self-approved.
  • Filed as two plans on purpose, so scope can be approved by approving one and not the other. browserless and neko have different second-user cases and materially different blast radii; bundling them would hide that.
  • browserless passes the second-user test on a fact, not a hope. UIS already ships Uptime Kuma; its real-browser monitor type calls chromium.connect(remoteBrowser.url) and the image UIS deploys has no Chrome — so that monitor type is dead in every installation today. browserless is the missing piece, and serves the exact Playwright websocket route Uptime Kuma connects to. Both facts verified in upstream source.
  • A related gap this surfaced: UIS's "E2E tests" are all API-level. Nothing can currently assert that a page renders, for any of the ten-plus services UIS ships with a web UI. OBS-F6 — the flaky Grafana E2E — is a test asserting on API responses because asserting on the rendered dashboard was impossible.
  • neko is argued both ways in its plan and gated hard. It fails the purpose test in the environment UIS calls baseline: on Rancher Desktop the human is already where the cluster is, with a logged-in browser open. Recommended shipped-but-never-default, because the Case-2 auth problem is real and UIS has no other answer — but nothing establishes a second installation wants it.
  • Proposes a new AUTOMATION category (manifests 400–429, 430–499 reserved). OBSERVABILITY was tempting and is wrong: browserless is a capability observability consumes, not an observability tool.

What changed 2026-08-21​

  • New investigation landed and closed the same day: roaming-dependency-addresses — not ranked, nothing to investigate next. Arrived as a productisation request from ops for a running ollama-endpoint-manager, and was answered rather than deferred: LiteLLM partly covers it. Ordered address preference (order) and health-check-driven routing are native; the fail-fast property is not, and upstream has explicitly chosen the opposite ("All deployments marked unhealthy by health checks, bypassing health filter" in router.py). Verdict: not an AI/ML component — it belongs to the dependency layer, as an optional rendering of dependencies-shim-services Phase 1.
  • Decided the same day (Terje): the reconciler is installation implementation, not UIS. No service, no name, no dependency-layer rendering. A UIS feature needs a second plausible user, not just a generalisable mechanism, and two roaming sleeping Macs is a single-installation topology — the boundary both repos already state. Reproducing it is ops' work, in the home repo. The challenge this raised against dependencies-shim-services is withdrawn; that plan is unaffected and the candidate-list-vs-discovery distinction is parked in its notes rather than left open.
  • What survives is entirely about LiteLLM, not about the reconciler: the parity findings, and one unwritten child — a LiteLLM recipe (two deployments, one model_name, order: 1/order: 2, enable_health_check_routing, with an honest account of the probe cost and the fail-fast caveat). That is general and has obvious second users. It is the only live thread left in that file, and the reason it sits in backlog/ rather than completed/.
  • Evidence that static Endpoints drift is not hypothetical. Both Ollama addresses committed in hosts/asgard/ollama-backends.yaml are stale; read live on asgard the same day, both hosts had moved and neither committed address was in use. The reference installation only kept working because an unproductised reconciler was absorbing it.
  • version-pinning (Tier 2 #9) gained a concrete consumer. The LiteLLM chart is unpinned, and order — the mechanism the recipe half of the verdict depends on — was broken in v1.80.11 (BerriAI/litellm#18444) and silently degrades to round-robin across a dead address when it regresses.

Tier 0 — in flight​

INVESTIGATEs that still live in backlog/ because their work isn't fully shipped yet — either active on a feature branch, or investigation-complete and waiting for a child PLAN to be drafted. No fresh investigation work needed; listed here so the priority view surfaces what's already moving. Items whose child PLANs have all shipped are not listed — they're in completed/.

InvestigationChild plansState
system-observability6 in backlog/ (001 log-collection, 002 alert-baseline, 003 service-dashboards, 004 external-targets, 006 service-probes, grafana-deploy-gate-fix)The largest thing in flight — and now merged, but not finished. 55 tasks done / 56 open. Everything through 2026-08-13 is on main: OBS-F6 is closed and verified against the shipped container image, not a patched one. What remains is what the platform's own requirement turns on — 003's dashboards are still applied by hand, so a rebuild from nothing does not produce them. Also open: Loki retention undecided (001), cert expiry (002), external-targets phases 2–4, grafana Phase 4 (move E2E to a verify playbook), and 006 at 0/27. Do not open 006 before the artifact convention is decided — 003 and 006 must share one, and deciding it unblocks both plus Tier 3 #17.
service-uptime-kuma2 in active/, 2 in backlog/, 1 shippedDeployed on assist with 19 monitors reconciling. Its own 002 plan records the gap that matters: nothing notifies — no job calls the push URLs. Do not close this investigation until that is either done or split out.
service-litellm-install-reliability3 in backlog/ (002 version-pinning, 003 undeploy-purge, 004 config-portability), 1 shipped001 schema-verify shipped; the remaining three are drafted and unstarted.
system-platform-provisioning-layer1 in backlog/ (aks-001b manual-setup), 2 shippedStatus ACTIVE, AKS-focused. Step 1 verified end-to-end 2026-05-11 against a real Azure subscription; platforms/azure-aks/ is the production path. Next concrete work: Step 2 (start/stop/scale so the cluster doesn't bill 24/7).
cli-stack-profiles1 in backlog/ (grafana-optional-datasources)The child plan is blocking: no --profile below full can install until it ships, and uis stack install observability --skip-optional is broken today (CLI-F1).
service-authentik-user-confignone draftedInvestigation Complete — Ready for PLAN. Unblocks Tier 3 backstage-auth.
system-remote-deployment-targetsnone draftedInvestigation Complete. Child PLAN drafting next; do not open as new investigation work.

Tier 1 — do next (load-bearing or unblocks active work)​

#InvestigationEffortWhy this tier
NEWtemplates-multi-surface-applicationS–MFiled 2026-09-07 from the first real application tenant (urb-agents#159). Ranked Tier 1 for two independent reasons. First, there is a person waiting: a front-end is being built against this tenant's API, so the install being four hand-sequenced steps is on someone's critical path. Second, TPL-F3 is a latent defect regardless of this tenant — template.sh:405 calls uis deploy "$svc" with no --app while the configure call two lines below passes --app, so the two halves of one loop disagree about whether a service is multi-instance; any multi-instance service is unusable from a template today. ⚠️ This is option A and it must stay option A. Do not let it grow into a UIS Application type — ANALYSIS-nais-uis §4 item 13 defers that behind items 1, 3 and 5, none of which have landed, and §4's own note is that building the declaration before the capabilities is the way to get a half-built abstraction. The mechanism to extend already exists and is called a template. TPL-Q3 (may an app ship its own template from its own repo?) needs a product ruling from Terje, not a design — it means executing a deploy plan, and piping an init file into psql, from an arbitrary repository.
NEWsystem-topology-coverageMFiled 2026-08-26 from an ops requirement. Three defects in one morning, all the same class: an assumption true only of the development topology. The tester and production differ on exactly two axes — context name and postgres shape — and all three defects lived there. Ranked Tier 1 because it is the systemic version of a failure that has already cost four hand-found production defects, and because part of the answer is cheap: a lint, not a second cluster. Outcomes 1 and 4 shipped; outcome 2 (exercise the proxy topology) is open. ⚠️ Do not "fix" this by switching the tester to production's shape — in-cluster postgres is a supported configuration and swapping moves the blind spot.
NEWThe oauth2-proxy OAuth app is owned by a person, not an orgSFiled 2026-09-18 from the tester's consolidated report. The GitHub OAuth app now gating a public host is registered under an individual account. A personal OAuth app cannot be rotated by anyone else and does not transfer, so the credential behind a production login gate has a single point of failure that is a person rather than a system. 🔵 Nothing is broken and nothing is urgent — the gate works and the app is correctly configured (Ov23li prefix confirms an OAuth App rather than a GitHub App, one callback URL, wildcard matching off). This is the succession half: who can rotate the secret after an incident, and what happens when the owner is unavailable. ⚠️ Re-registering is not free — a new app means a new client id and secret, one ./uis secrets apply, and a gate roll, so it is cheapest to do while exactly one host is gated, which is the situation today. The recommendation before the app was created was org-owned; it is worth confirming that was a decision rather than an oversight.
NEWundeploy leaves the service in autostart, and one stale entry stops the whole bootSFound 2026-09-18 during the oauth2-proxy gate round; the tester inferred it from a printed line, and reading the code made it worse than they claimed. cmd_deploy calls enable_service on every successful single-service deploy (provision-host/uis/manage/uis-cli.sh:504) and cmd_undeploy never calls disable_service — the only callers are cmd_disable and the toggle in lib/service-auto-enable.sh:194. So undeploy any service and a later bare ./uis deploy brings it back, which is already wrong. The sharp edge is what happens when the re-deploy fails: deploy_enabled_services does return 1 # Stop on first failure per design (lib/service-deployment.sh), so one stale entry aborts the boot and every service after it in the list never deploys. The gate is the concrete case — undeploy it, leave protected: empty, and its deploy now refuses by design, so a bare uis deploy stops there. ⚠️ Not fixed in 1.6.125 deliberately: this is shared-CLI behaviour for every service, it has a symmetry question in it (should undeploy disable, or should deploy stop auto-enabling?), and it does not belong inside a gate fix. 🔵 Neither half is measured yet — the tester did not run bare uis deploy to confirm the abort, and nor have I. Reproduce both before choosing the fix.
NEWTraefik v3 — declare it, assert it, and keep the managed install in parityS–MFiled 2026-09-18 from Terje's decision above. Two pieces, both small, neither done. (1) Assert the running version before applying v3 CRDs, and refuse with the reason. Today nothing checks, and #199's failure mode is the worst kind: v3 manifests on a v2 controller 404 every service while the playbook exits 0. (2) Keep the UIS-managed install in parity — 003-setup-traefik.yml pins chart 39.0.7 to match what rancher-desktop's k3s ships, and the playbook's note that both sides "migrate together as one coordinated change" is a comment rather than a check, so the two can drift silently. ⚠️ Ranked Tier 1 because a tenant is waiting: per-path auth policies need PathRegexp, which is v3-only, and a service being migrated onto UIS gates restricted pages by a path pattern today. Without the assertion we would be building on an assumption the platform does not verify. 🔵 Related but not the same as Tier 2 #9 (version-pinning) — that one is about image tags across services; this is a minimum platform version with no enforcement. 🔴 2026-09-18, and this is now measured rather than argued. The gate's PathRegexp route was tested on a cluster running Traefik 3.7.4 and worked, with no matcher complaint in the log. That is one host agreeing, not an assertion — and the same host was on Traefik 2.10.5 a week earlier, after a Rancher Desktop rebuild silently fell back to bundled k3s v1.25.16. It has flipped v3 → v2 → v3 in eight days, without anyone choosing that. On the v2 leg the route would 404 with exit 0 and nothing in the log. So the failure mode is not hypothetical and the downgrade is not operator error: it arrives with a rebuild.
1secrets-template-defaults-claritySFoundational fix to the secrets workflow every service depends on. The current silent-overwrite confusion between 00-common-values.env.template and default-secrets.env produces bug reports from contributors and slows every onboarding. Investigation already half-shipped via the existing template scaffolding; closing it out is a small read-and-decide.
2verification-playbooks-usageMPromoted from Tier 2 on evidence, and the evidence keeps arriving. Written 2026-03-12, it states the risk as "verification playbooks present but not wired into active setup or test flows → deployments report success when no real validation happened." Three separate confirmations since: (1) 031-test-alloy.yml was reachable by no command at all while its docs page told users to run it; (2) uptime-kuma's verify was invisible to test-all; (3) kubectl run --rm -i silently returns rc=0 with empty stdout when the container outlives the attach — every until: asserting on stdout then reads a successful call as a failure. (3) is the serious one: the same idiom appears across other services' verify playbooks, so an unknown number of them can fail or pass for reasons unrelated to the service. All three are fixed ad-hoc; this investigation is the systematic version. Stays Tier 1, not Tier 0, on purpose — its child plans are point fixes to the services that happened to break, not the investigation's output.
3external-or-in-cluster-servicesMNew 2026-08-13. OpenBao, the registry cache and the backup chain run on Odin, built by hand, with no service definition — the exact state Alloy was in before it was made a real service. Rebuild Odin and they are rebuilt from memory; OpenBao holds the vault recovery keys. The requirement is that they also run on Rancher Desktop, which needs one convention: how a service is provided externally in one installation and deployed in-cluster in another, behind one interface. .uis.extend/ is the half-built precedent (shipped twice, for watching external things, never for substituting them), and six database services already live this shape undeclared. A full pct list of Odin found six components, not three: pg and minio already have both a service definition and a hand-built proxy (unproductised, not missing), bao and the registry mirrors have nothing, and nas (Samba/NFS) is parked pending a scope decision — the laptop answer is probably Rancher Desktop's existing storage class rather than shipping Samba, but that should be decided deliberately. ops is not a service: it is the host running uis-provision-host itself. First child plan SHIPPED 2026-08-14 — PostgreSQL and MinIO both run from the convention, and the reference installation has zero hand-written proxies left. Remaining children: bao, the registry mirrors, cluster backup. Decide alongside the observability artifact convention — same question in a different costume, and two answers would never converge.
4service-openbaoMNew 2026-08-14. OpenBao holds the vault recovery keys and is the only Odin component with neither a service definition nor an investigation — built entirely by hand, so a rebuild reconstructs it from memory. Written expecting it to reuse the proxy convention; measuring showed the convention does not fit. OpenBao speaks TLS with a private CA whose SANs cover no in-cluster name (BAO-F2); its kubernetes auth runs into the cluster via TokenReview, a direction a proxy cannot help with (BAO-F3); and nothing talks to it except ESO — apps read ordinary Secrets — so the interface needing parity is the ClusterSecretStore name, not an address (BAO-F1). That makes dev cheaper than expected: run it in-cluster under the same store name and every ExternalSecret works unchanged.
5secrets-dev-to-productionS–MNew 2026-08-14. Follows one secret from a developer's laptop into production. The application side is already portable — the manifest is identical — but the machinery meant to notice a missing or fake value is not. envsubst renders an unset key as an empty string, silently (verified); validation is a hardcoded allowlist of 7 required and 6 weak-value names against a 62-key template, so a newly added secret is checked for neither; and validate_secrets is not on the uis deploy path at all. Net effect: LocalDevStripe123 can reach production without a single warning. Same allowlist-that-new-things-never-join shape as the verify-registration defect. Do this before the vault question — none of these failure modes is fixed by having a vault.
6uis-deploy-no-playbook-semanticsSGenuine ambiguity in the deploy code today: services with SCRIPT_PLAYBOOK="" produce undefined behaviour. Affects every "metadata-only" service. Investigation gap is small; pinning the contract removes a class of latent bugs across all current and future services.
7uis-deploy-auto-regen-secretsMUX gap that bites tester loops repeatedly: stale kubernetes-secrets.yml produces silent failures. Decisions here lock down idempotency for the whole deploy command. Tester-feedback-driven; high payoff per hour.

Tier 2 — do after Tier 1 (independent, ready, valuable)​

#InvestigationEffortWhy this tier
8in-cluster-port-on-servicesSSmall services.json schema addition that downstream consumers (Backstage catalog, docs generator, future MCP integrations) keep working around. Cheap to land; immediate downstream payoff.
9version-pinningMCross-cutting consistency review: which services have pinned image tags vs :latest. Affects supply-chain hygiene and CI reproducibility. Independent of other tiers; ships value on its own.
10service-version-metadataMTied to docs/CLI display: how service scripts expose version info. Closes a presentation gap visible on every service page. Pairs naturally with #6 if done together.
11system-backup-and-schedulingMNewly triaged; scope narrowed 2026-08-13. There is no Velero anywhere. Odin runs five mechanisms across three layers — vzdump, sanoid, syncoid, restic, pgBackRest — none Kubernetes-aware and none able to run on Rancher Desktop. So this investigation owns the host layer only: documenting and making reproducible what already exists, outside UIS where it belongs. Cluster backup (namespaces, PVCs, secrets) is a separate, currently-uncovered gap — vzdump restores the whole k8s VM but not one namespace — tracked under external-or-in-cluster-services EXT-F6. Original note: Backup + scheduled-job story for the platform. Load-bearing in the same way #2 is: an unverified backup and an unwired verify playbook fail identically — silently, and only when you need them. Ranks below Tier 1 only because no current work is blocked on it.
12system-registry-cacheMNewly triaged. Pull-through image cache. Independent, self-contained, and pays off on every cluster rebuild — which the platform does often by design.
13uis-connect-commandsMUser-facing convenience: uis connect <service> opens an interactive client without requiring host-side tooling. Independent of platform/deployment tiers; shippable as a self-contained slice.
14cli-grammar-harmonizationMNewly triaged. Consistency of the uis <verb> <noun> surface. Related in spirit to #2: the verify command exists in two grammatical forms (uis verify alloy and uis alloy verify), which is precisely how the registration defect stayed hidden. Worth doing before the CLI surface grows further.
15cli-top-level-docSNewly triaged. One coherent top-level CLI reference. Small, and the 2026-08-11 help-text gaps (three services dispatching correctly but undocumented) show the current help is maintained by hand and drifts.
16docs-markdown-update-logicMImproves the docs-generation pipeline so metadata-driven sections update without overwriting manual prose. Quality-of-life for contributors maintaining service pages.
17cli-network-export-importMNewly triaged. Moving network config between installs. Ready but not blocking anything; sits at the bottom of Tier 2 until a concrete consumer pulls on it.

Tier 3 — defer until prereqs ship​

These have known prerequisites that are still open. Don't open them yet — the prereq's outcome materially changes the investigation's scope.

#InvestigationWaits onWhy defer
18backstage-authauthentik-user-config (Tier 0, ready for PLAN)Adding Authentik OIDC to Backstage assumes Authentik's user-config story is settled. Open the auth investigation only after the user-config PLAN has shipped, otherwise the OIDC client config keeps shifting.
19provision-host-tools-and-authplatform-provisioning-layer (Tier 0, in flight)Decisions about which CLIs (Azure / AWS / GCP / Terraform) live inside uis-provision-host depend on the platforms model that the active feature branch is still settling. Designing tool-install + auth-state before platforms lock in = rework.
20monitor-definitions-with-servicesobservability 003 + 006 (Tier 0)Newly triaged. Shipping monitor/probe definitions alongside each service shares an artifact convention with observability's dashboards (003) and probes (006). Both are open and still deciding that convention; designing a third consumer first guarantees rework.
21migrate-hosts-to-platformsplatform-provisioning-layer (Tier 0, in flight)Includes the documentation migration, folded in deliberately — the code and docs sides belong together. Waits on the same platforms model as #16.
22tailscale-cross-cluster-backboneplatform-provisioning-layer; a real second siteNewly triaged. Cross-cluster networking is only designable once there is a second cluster that must be reached the hard way. Related to the Oslo+Bodø multi-site intent, which is not yet built.
23dct-argocd-deployargocd as a stable UIS serviceThe "deploy from inside DCT with one command" flow needs argocd to be the deployment substrate. ArgoCD has a manifest in UIS but isn't an everyday service yet; investigate this once argocd is operationally normal.
24enonic-deploymentenonic-as-stable-serviceCovers both apps (JAR pipeline — chosen-approach decided) and content (still open). Pull-based deployment design assumes Enonic XP is operationally stable in UIS. Merged from the two earlier app + content investigations on 2026-05-15.
25email-smtp-serviceproduct clarity (which services need email first?)Cross-cutting platform service. Worth opening only when the first concrete consumer (Authentik password resets? a notification path?) is actually pulling on it.

Tier 4 — ideas, not investigations​

These are sketches / parking-lot entries, not concrete research targets. Don't open them as INVESTIGATEs — let the surrounding context resolve, then either promote to a real INVESTIGATE or delete.

#ItemWhat to do
26espocrmCurrently four URLs and zero analysis. Either promote to a real INVESTIGATE (with a goal + comparison against alternatives) or delete.
27dagster✅ Trigger met 2026-08-22 — no longer deferred. This row said to wait for "the data-orchestration use case to materialise into a real consumer". It has: Terje decided Dagster becomes a reusable UIS service, and Atlas is done and waiting — 41 sources, 40 Pipes-enabled, code-location image publishing on every commit. Shipped 2026-08-23 as PLAN-service-dagster-001-deploy — approved, built, and independently verified after three rounds and six defects. The investigation stays here as the design record; it is four months stale on source counts but its structural decisions held.
28metabaseSimilar to #24 — internal BI / data exploration tool selection. Hold until there's a concrete first consumer driving the requirements.
29customer-onboarding-databaseNewly triaged. Self-described as "Draft / not yet scheduled." Leave as a sketch until there is a real customer-onboarding flow to design against.

Ready to move to completed/​

Every declared child PLAN has shipped. Per rule 4 these should be git mv'd and their rows deleted from this doc — listed rather than moved because moving files is a git operation.

InvestigationChild plan that shipped
service-backstage-enhancementsPLAN-004-backstage-api-entities
templates-first-uis-templatePLAN-002-uis-template-command

Housekeeping found during this refresh​

  • The parent-link key is not consistent, so plans go missing from any triage that walks it. PLANS.md's template says **Investigation**:, but PLAN-system-observability-006-service-probes legitimately has two parents and writes **Investigations**: (plural). Two parents is a real case the template does not cover — the convention should name both keys rather than the plural form being treated as a mistake.
  • PLAN-service-uptime-kuma-005-ship-the-pipeline genuinely has no parent link — it declares Prerequisites and Related but never its investigation. It is also ## Status: Active while sitting in backlog/, making it the seventh plan claiming Active.
  • Six plans carry ## Status: Active while PLANS.md says 1–2 at a time, and four of those sit in backlog/ rather than active/. The folders no longer describe reality — worth a pass before starting anything new.

Cross-cutting notes​

  • Two natural workstreams: UIS-internal correctness (Tier 1) and developer-experience polish (Tier 2, items #5–#14). They can run in parallel — different files, no merge contention.
  • The verification thread: #2 (verification-playbooks-usage), #11 (cli-grammar-harmonization) and #12 (cli-top-level-doc) are three views of one problem — the CLI surface and its test wiring are maintained by hand and drift silently. Doing #2 first gives the other two concrete evidence to work from.
  • Backstage cluster: Backstage core (shipped) → #15 (auth) → enhancements (ready to move to completed/).
  • Platform/host cluster: Tier 0 platform-provisioning-layer (in flight) → #16 (provision-host tools) → #18 (host migration) → #19 (cross-cluster backbone). Tight chain; resolve in order.
  • Observability cluster: Tier 0 system-observability is the bottleneck for #17 (monitor definitions) and shares an artifact convention with it. Finish 003/006 before opening #17.
  • External coupling with atlas: atlas's deployment-pipeline investigation explicitly waits on UIS's dagster signal. Resolving #24 unblocks atlas's deployment-pipeline; deferring it keeps that block in place — fine if no UIS consumer is pulling on dagster yet, but worth flagging when atlas next asks.
  • Idea-vs-investigation ratio: 4 of 35 are still ideas (Tier 4). Healthy — most of the backlog is concrete work, not brainstorm residue.
  • Investigation-completion debt: 2 of the 7 Tier-0 entries (remote-deployment-targets, authentik-user-config) are "investigation complete" without a child PLAN drafted. Picking one up as the next PLAN-drafting task closes more uncertainty than starting any Tier-1 investigation.

How to use this doc​

  1. Pick the top unstarted item from Tier 1; if all of Tier 1 is in flight or done, move to Tier 2.
  2. When starting an INVESTIGATE, leave it in this folder and update its Status: line to note the work is in flight.
  3. When an INVESTIGATE produces a recommendation and a child PLAN is drafted, update this doc: move the row to Tier 0 and note the PLAN it spawned.
  4. When every child PLAN of an INVESTIGATE has shipped, git mv the file to completed/, fix any cross-references, and delete its Tier 0 row from this doc — completed/index.md carries it from then on.
  5. When a Tier-3 prereq lands, promote its dependents up to Tier 2 in the next refresh.
  6. Re-rank quarterly or after every 3 INVESTIGATEs ship — whichever comes first.

Keeping it current​

The May→August drift happened because rules 3 and 4 fire at moments when the person is thinking about code, not about triage. Concrete triggers, so the update is a step in the work rather than a separate chore:

  • A new INVESTIGATE-*.md lands in backlog/ → add its row in the same commit. An investigation absent from this doc is invisible to triage.
  • A PLAN moves to completed/ → check whether its parent investigation has any child plans left. If not, move it and delete its row (rule 4).
  • A PLAN is drafted → move its parent to Tier 0 (rule 3).
  • A new PLAN is written → give it a **Investigation**: header linking its parent, or it will not be findable from here.
  • The counts in the header (N INVESTIGATE files) are checkable against ls backlog/INVESTIGATE-*.md | wc -l — if they disagree, the doc is stale.