dave
cdc6bb33d3
Completes 109 of 121 tasks. Every remaining task needs a tenant connection
(T055, T056, T101-T103) or an Azure Automation account (T115-T121).
354 offline Pester tests PASS
Engine purity (Principle IV) PASS
Sanitization (SC-013) PASS (156 files)
Graph module loaded in tests none (SC-008 holds)
What landed
- Four-layer configuration validation with stable finding codes, covering
every VR-002 and VR-003 condition, plus a 23-fixture invalid-config corpus
- Run loop, audit records (NDJSON through a single sink), summaries,
reconciliation, and exit codes 0-6
- Persistence behind a single write-body builder whose result always has
exactly one key
- Invoke-PersonaEngine.ps1 and Edit-PersonaEngineConfig.ps1
- Six docs, two pipelines, traceability matrix, V-5a and sanitization records
Three deviations from tasks.md, each recorded in its status block
T033 is not in Resolve-UserPersona. evaluationErrorThreshold is run-level
state and the rule engine is pure; a counter there would break Principle IV.
It lives in New-PersonaRunCounter and is applied in the run loop.
A new src/Engine/ layer holds Invoke-PersonaEngineRun. The entry script
imports the manifest, which requires Microsoft.Graph.Authentication, so a
loop living only inside it could not run on a machine without the Graph SDK
and SC-004 could not be proven at all. The entry script is now a thin
wrapper and what ships is what is tested.
The invalid-config corpus is generated by a committed script, with the
generated fixtures committed too, so a reviewer sees the fixture in the diff.
Defects found by running the code, not by reading it
Group and role ID lists were double-wrapped: @(Get-PersonaGroupIdPage ...)
around a comma-returned array collapsed every membership list into one
bogus space-joined entry. That is a silent false non-match, exactly what
FR-013 exists to prevent.
A 403 whose status appears only in the exception message parsed as $null,
which the retry policy treats as a transport error - five requests per
account against a tenant already refusing. Status extraction now falls back
to the message text, bounded to 400-599.
The sanitization scan walked tracked files only, so it covered 34 of 156
files and none of this phase's code. It now scans untracked non-ignored
files too, and a negative control confirms it catches a planted leak.
Test-Json reports one error per violating location, not first-failure-only
as the V-5a draft claimed. Record and pin corrected.
Enforcement remains blocked on the V-4 security sign-off.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
214 lines
8.4 KiB
Markdown
214 lines
8.4 KiB
Markdown
# Quickstart: Persona Engine Validation
|
||
|
||
**Date**: 2026-08-20 | **Spec**: [spec.md](spec.md) | **Plan**: [plan.md](plan.md)
|
||
|
||
Runnable scenarios that prove the feature works. Ordered by the build sequence in
|
||
[plan.md](plan.md) — each stage is validatable before the next exists. Scenarios 1–3 require no
|
||
tenant, no credentials, and no network.
|
||
|
||
Structural details live in [data-model.md](data-model.md) and [contracts/](contracts/); this guide
|
||
does not repeat them.
|
||
|
||
## Prerequisites
|
||
|
||
**Current constraint**: no Azure Automation account. Everything runs locally on PowerShell 7 with
|
||
user accounts and delegated authentication. Scenario 5 is deferred; Scenarios 1–4 and 6 are all
|
||
available now.
|
||
|
||
| Scenario | Stage | Requirement |
|
||
| --- | --- | --- |
|
||
| 1–3 (offline) | A1 | PowerShell 7.4, Pester 5.x, PSScriptAnalyzer. No tenant, no network. |
|
||
| 4 (read-only) | A2 | An app registration with admin consent for the three delegated scopes, and a **non-privileged** user account to sign in with. Closes V-1 (read), V-3. |
|
||
| 5 (automation) | B | **Deferred** — no Automation account available. Closes V-3b, V-5b when it lands. |
|
||
| 6 (enforcement) | A3 | Delegated `User.ReadWrite.All`, **written security sign-off (V-4)**, reviewed `-WhatIf` evidence, and **purpose-created test accounts** as the write targets. |
|
||
|
||
### Local connection
|
||
|
||
```powershell
|
||
Connect-MgGraph -Scopes 'User.Read.All','GroupMember.Read.All','RoleManagement.Read.Directory'
|
||
```
|
||
|
||
Sign in as an ordinary user account, not a Global Administrator — see the caution in Scenario 4.
|
||
|
||
---
|
||
|
||
## Scenario 1 — Rule engine determinism, offline
|
||
|
||
Proves SC-001, SC-003, SC-008 and the `Unknown` propagation table in
|
||
[data-model.md](data-model.md).
|
||
|
||
```bash
|
||
pwsh -NoProfile -Command "Invoke-Pester ./tests/RuleEngine -Output Detailed"
|
||
```
|
||
|
||
**Expected**: all pass with no network access. Specifically:
|
||
|
||
- Every synthetic user yields exactly one outcome.
|
||
- Shuffling fixture order changes nothing.
|
||
- A rule matching at priority 10 wins over one matching at 20, and evaluation stops.
|
||
- A `MembershipRecord` with `RetrievalSucceeded = $false` yields `EvaluationError`, never a
|
||
non-match.
|
||
- Depth beyond `maxConditionDepth` is rejected rather than silently truncated.
|
||
|
||
**Disconnect the network and re-run.** Identical results, or SC-008 is not met.
|
||
|
||
---
|
||
|
||
## Scenario 2 — Configuration validation
|
||
|
||
Proves SC-009, SC-010 and the four-layer ordering.
|
||
|
||
```bash
|
||
pwsh -NoProfile -File ./Edit-PersonaEngineConfig.ps1 -ConfigPath ./config/persona-engine.example.json -ValidateOnly -NonInteractive
|
||
```
|
||
|
||
**Expected**: exit code `0`, no findings.
|
||
|
||
Then run the invalid-configuration corpus in `tests/TestData/InvalidConfigs/` — one file per VR-002
|
||
and VR-003 condition:
|
||
|
||
```bash
|
||
pwsh -NoProfile -Command "Invoke-Pester ./tests/Configuration -Output Detailed"
|
||
```
|
||
|
||
**Expected**: each file produces its documented finding code, severity, and location; blocking
|
||
findings return a non-zero exit code with **no prompt and no hang** (SC-010).
|
||
|
||
---
|
||
|
||
## Scenario 3 — Safety invariants
|
||
|
||
Proves SC-004 and SC-005 with the write adapter mocked. This suite is the reason the persistence
|
||
adapter is built last.
|
||
|
||
```bash
|
||
pwsh -NoProfile -Command "Invoke-Pester ./tests/Safety -Output Detailed"
|
||
```
|
||
|
||
**Expected**:
|
||
|
||
- Full synthetic population under `-WhatIf`: write adapter call count is exactly `0` — asserted, not
|
||
inspected (SC-004).
|
||
- Every captured request body has exactly one key, equal to `engine.targetAttribute` (SC-005).
|
||
- `New-PersonaWriteBody` throws for any other attribute name and for an attribute absent from
|
||
`approvedWritableAttributes`.
|
||
- A `-Debug` run **without** `-WhatIf` still reaches the write path — `-Debug` is not a safety
|
||
control.
|
||
|
||
---
|
||
|
||
## Scenario 4 — Read-only tenant preview (User Story 1)
|
||
|
||
The first connected run. Uses a read-only identity, so it is safe by construction rather than by
|
||
correct behaviour.
|
||
|
||
> **Sign in as a non-privileged account.** V-3 asks whether the three scopes are *sufficient*.
|
||
> A Global Administrator answers yes regardless — the scope narrows the token, but the account's
|
||
> directory roles still grant broad read access, so the run succeeds whether or not the permission
|
||
> set is correct. Running this as GA produces a green result that means nothing.
|
||
|
||
```bash
|
||
pwsh -NoProfile -File ./Invoke-PersonaEngine.ps1 -ConfigPath ./config/persona-engine.json -WhatIf -Verbose
|
||
```
|
||
|
||
**Expected**:
|
||
|
||
- A result line appears for every in-scope user, visible before the next user is processed (SC-012).
|
||
- Differences report as `WouldUpdate` with stored value, calculated value, and matched rule ID.
|
||
- Interim summaries at the configured interval; a final summary always; reconciliation passes at
|
||
every summary (SC-007).
|
||
- Zero write requests — confirm independently in the Entra sign-in and audit logs, not only from
|
||
console output.
|
||
|
||
Single-user check first, before the full population:
|
||
|
||
```bash
|
||
pwsh -NoProfile -File ./Invoke-PersonaEngine.ps1 -ConfigPath ./config/persona-engine.json -UserObjectId <ACCOUNT-OBJECT-ID> -WhatIf
|
||
```
|
||
|
||
**Also close V-1 here**: run the single-user check against a cloud-only user, a currently-synced
|
||
user, and a formerly-synced user. Confirm the persona extension **reads** on all three. The write
|
||
half of V-1 is closed in Scenario 6.
|
||
|
||
**Idempotence check** (SC-002): run twice unchanged. The second run reports the same counts and zero
|
||
additional proposed changes.
|
||
|
||
---
|
||
|
||
## Scenario 5 — Azure Automation PowerShell 7 *(deferred — Stage B)*
|
||
|
||
**Not runnable in the current stage.** No Automation account is available. Recorded here so it is
|
||
not lost, and so the Stage B entry cost stays visible.
|
||
|
||
Proves NFR-001, NFR-008 and closes V-3b and V-5b.
|
||
|
||
1. Import the module and publish the runbook with the schedule **disabled**.
|
||
2. Run the runbook with `-WhatIf` using the managed identity.
|
||
3. Record the runtime's exact PowerShell version.
|
||
4. Confirm `Test-Json -SchemaFile` behaves as observed locally in V-5a — its error-reporting
|
||
behaviour varies by version, and the layer-2 wrapper depends on it (OTD-005). **Run this first**;
|
||
it is the cheapest item with the highest chance of surprising you.
|
||
5. Confirm the three read scopes work as *application* permissions on the managed identity (V-3b).
|
||
|
||
**Expected**: the run completes with only `Microsoft.Graph.Authentication` imported, and output
|
||
matches the equivalent local `-WhatIf` run.
|
||
|
||
**Until this scenario passes, v1 is not complete.** The Definition of Done requires an Automation
|
||
PowerShell 7 run. Stage A completion is a real milestone and worth claiming — but it is not v1.
|
||
|
||
---
|
||
|
||
## Scenario 6 — Enforcement (User Story 8)
|
||
|
||
**Gated.** Do not run until all of these hold:
|
||
|
||
- [ ] V-4 security sign-off on the OTD-003 compensating controls, in writing
|
||
- [ ] `-WhatIf` impact evidence from Scenario 4 reviewed and approved
|
||
- [ ] Scenarios 1–5 passing
|
||
- [ ] Kill switch and rollback procedure documented
|
||
- [ ] OTD-001 – OTD-005 closed
|
||
|
||
```bash
|
||
pwsh -NoProfile -File ./Invoke-PersonaEngine.ps1 -ConfigPath ./config/persona-engine.json
|
||
```
|
||
|
||
**Expected**:
|
||
|
||
- Only changed values are written; unchanged users produce no request (SC-002).
|
||
- Every write body contains exactly one attribute (SC-005).
|
||
- Every `Updated` audit record carries `previousValue` — without it, rollback is impossible
|
||
retroactively (OTD-010).
|
||
- `EvaluationError` users are skipped with their stored persona intact (FR-014).
|
||
|
||
**Closes V-1 (write half)** and **V-2**: confirm the dynamic membership group built on the persona
|
||
extension populates, and that a Conditional Access policy assigned to that group applies.
|
||
|
||
---
|
||
|
||
## Verification item coverage
|
||
|
||
| Item | Closed by | Available now? |
|
||
| --- | --- | --- |
|
||
| V-1 | Scenario 4 (read) + Scenario 6 (write) | Yes |
|
||
| V-2 | Scenario 6 | Yes |
|
||
| V-3 | Scenario 4, as a non-privileged account | Yes |
|
||
| V-3b | Scenario 5 | **No — Stage B** |
|
||
| V-4 | Out-of-band security review — **gate on Scenario 6** | Yes (a conversation, not a tenant) |
|
||
| V-4a | Investigation; no scenario | Yes |
|
||
| V-5a | Scenario 2, behaviour pinned in a unit test | Yes |
|
||
| V-5b | Scenario 5 | **No — Stage B** |
|
||
|
||
## Exit code check (SC-011)
|
||
|
||
Every documented exit code must be reachable. Cover them deliberately rather than incidentally:
|
||
|
||
| Code | How to trigger |
|
||
| --- | --- |
|
||
| `0` | Scenario 4 |
|
||
| `1` | Any invalid configuration from Scenario 2 |
|
||
| `2` | Run with an unauthorized or expired identity |
|
||
| `3` | Fault injection on enumeration |
|
||
| `4` | Fault injection on a required data provider |
|
||
| `5` | Fault injection on the counter path (reconciliation defect) |
|
||
| `6` | Fault injection on an unhandled engine path |
|