T-066 — The dashboard, committed as JSON

Picking this up? Read CONTRIBUTING.md first, then claim the matching issue and work on a branch. Finished means all six definition-of-done gates, not five. If anything below disagrees with a contract, the contract wins — open a contract change issue instead of implementing either version. Update this task’s row in the ledger in the same pull request.

Milestone M6 · Estimate 30 min (build tiles in the console, export, wire one Terraform resource. Honest risk: hand-authoring eight tiles as raw JSON blows the budget — build in the console and export, that is the intended path.)

Preconditions — T-064 (SLO objects exist and are referenced by id), T-065 (alert policies exist, so the dashboard can annotate the same signals), T-060…T-063 (every metric queryable in Managed Prometheus). You inherit a fully alerted environment with no single screen to look at: diagnosing means retyping PromQL.

Goal — Commit one on-call dashboard as reviewable JSON in the repo, applied by google_monitoring_dashboard, with six required tiles laid out in doc 06’s 3 a.m. reading order plus the two optional ones if time allows.

1. Why this task exists

A dashboard edited in the console at 3 a.m. is lost work: unreviewable, unrestorable, and free to drift away from the metric names it queries. Doc 06 §6.8 therefore puts the JSON in the repo and ADR-009 D10 makes dashboards Terraform-managed but disposable — if this dashboard vanishes nothing breaks, which is exactly why it may be regenerated by export rather than hand-maintained. The load-bearing content is the order of the tiles, which encodes the diagnostic sequence.

2. Contracts to obey

What	Pinned by
Metric names, types, tag value sets in every tile query	docs/contracts/C4-observability.md#ct4-metrics
Metric type + prometheus_target resource form (prometheus.googleapis.com/<name>/<kind>)	#ct4-scrape, doc 06 #obs-empty
The two must-be-zero counters shown as counters, never as an SLO	#ct4-zero, doc 06 #obs-not-slo
Tile list, contents and reading order	doc 06 #obs-dashboard (authority)
Cardinality budget — no per-payout or per-key tag in any tile query	#ct4-cardinality
Terraform layout / project names	ADR-008 D1, C5-config-build-and-naming.md#ct5-naming

Precedence: if this spec and a contract disagree, the CONTRACT wins — stop and report (doc 04 §4.4).

3. Deliverables

Path	What
deploy/terraform/modules/observability/dashboards/dlock-oncall.json	The exported dashboard definition, pretty-printed, one key per line so a tile change is a readable diff
deploy/terraform/modules/observability/dashboard.tf	One google_monitoring_dashboard reading the JSON from disk; project id injected, not baked
deploy/terraform/modules/observability/dashboards/README.md	≤ 20 lines: how the JSON was produced (console → export), how to re-export, and the rule that the JSON is generated-and-committed, never hand-edited for anything but a query string
docs/06-observability-and-slo.md (modify, §6.8)	Correct the path shorthand observability/dashboards/ to the real path and the task id (§6.8 currently cites T-063)

4. Specification

Tiles, in this order — the order is the specification. Layout must be a grid read left-to-right, top-to-bottom; tile 1 occupies the full first row so it cannot be scrolled past.

#	Tile	Content	Chart type
1	Correctness counters	lock_fenced_out_total and rail_duplicate_attempted_total, 24 h sum	Two scorecards, large, threshold colouring so zero is green and any non-zero is red
2	SLO attainment + budget	S1 and S6 current 28-day attainment, minutes remaining, burn-rate sparkline	SLO/scorecard widgets bound to the T-064 SLO ids
3	Acquire outcome mix	lock_acquire_seconds_count rate stacked by outcome, split by backend	Stacked area
4	Acquire latency	p50 and p99 from histogram buckets, pg and etcd series side by side	Line, log-scale y if the pg/etcd gap flattens the chart
5	Payout journey	payout_execute_seconds_count by outcome, rail_submission_total by outcome, payout_backlog_age_seconds	Two stacked series + one line; one tile, so rail-vs-us is answered without a second look
6	Backend health	etcd leader changes and wal_fsync p99; Cloud SQL num_backends and replication lag for dlock-pg-lock	Line, two y-axes or two sub-tiles
7	(optional) Lock hold-time distribution	from lock.held_ms (§6.6)	Heatmap or p50/p99 line
8	(optional) Renewal-failure divergence	lock_session_lost_total vs lock_lease_expired_total on one chart	Line — the divergence is the signal

Each tile carries a one-line title that states the question it answers, in doc 06’s words (“Is this a money incident?”, “Errors, or just contention?”). A tile titled with a metric name forces the reader to re-derive intent; a tile titled with the question does not.

Production route: compose in the Cloud Monitoring console against live traffic (the load generator from T-045 is the honest source), confirm every tile renders non-empty, then export the definition and commit it. Strip console-generated ids/etags that would fight Terraform; keep dashboardFilters only if they are actually useful.

Terraform: exactly one google_monitoring_dashboard, JSON loaded from the file (templating limited to project id). No tile JSON inline in .tf.

5. Acceptance criteria

1. dashboards/dlock-oncall.json exists, parses as JSON, and is pretty-printed (multi-line).
2. It contains 6 to 8 tiles; tile order in the JSON matches the table above top-to-bottom.
3. Tile 1’s two scorecards reference lock_fenced_out_total and rail_duplicate_attempted_total and no SLO widget references either (§6.3.1).
4. Tile 2 references the SLO ids output by T-064, not a re-derived ratio query.
5. Every query string uses prometheus.googleapis.com/ metric types; grep for k8s_container in the JSON returns nothing.
6. No query contains a per-payout or per-key label (grep for payout_id, lock_key returns nothing) — #ct4-cardinality.
7. Every tile title is phrased as, or contains, the question it answers.
8. dashboard.tf contains no inline tile JSON; the file is read from dashboards/.
9. terraform plan shows exactly one dashboard resource to add; apply succeeds.

6. Verification

python3 -m json.tool deploy/terraform/modules/observability/dashboards/dlock-oncall.json > /dev/null && echo JSON-OK
grep -c '"title"' deploy/terraform/modules/observability/dashboards/dlock-oncall.json
grep -n 'k8s_container\|payout_id\|lock_key' deploy/terraform/modules/observability/dashboards/dlock-oncall.json || echo CLEAN
terraform -chdir=deploy/terraform/envs/dev plan -out=tfplan && terraform -chdir=deploy/terraform/envs/dev apply tfplan
gcloud monitoring dashboards list --project dlock-lab --format='table(displayName,name)'

Then, with the load generator running, open the dashboard and confirm every tile has data. A blank tile is the §6.1.1 failure, not a quiet system — fix the query, then re-export.

7. Out of scope

Tracing views and Trace-Explorer links (T-067), firing alerts to see them annotate the charts (T-068), benchmark comparison charts for publication (M7, T-070…075), and any new metric — if a tile needs data that does not exist, that is a C4 amendment, not a dashboard fix.

8. Hazards

The export/apply loop fights you if console ids and etags stay in the JSON — strip them or every apply shows spurious drift. Second: histogram percentiles must be computed from the ingested buckets, and a client-side percentile aggregated across pods is arithmetically meaningless (doc 06 §6.3) — do not average p99s across replicas in tile 4. Third: tile order is not cosmetic; putting latency before the correctness counters trains the on-call to debug performance during a money incident.

9. On completion

Mark the T-066 row done in tasks/README.md. Record whether tiles 7–8 landed, and note the doc-06 §6.8 path/task-id correction you made.