Implement Stage A: rule engine, validation, audit, and safety gates
Completes 109 of 121 tasks. Every remaining task needs a tenant connection
(T055, T056, T101-T103) or an Azure Automation account (T115-T121).
354 offline Pester tests PASS
Engine purity (Principle IV) PASS
Sanitization (SC-013) PASS (156 files)
Graph module loaded in tests none (SC-008 holds)
What landed
- Four-layer configuration validation with stable finding codes, covering
every VR-002 and VR-003 condition, plus a 23-fixture invalid-config corpus
- Run loop, audit records (NDJSON through a single sink), summaries,
reconciliation, and exit codes 0-6
- Persistence behind a single write-body builder whose result always has
exactly one key
- Invoke-PersonaEngine.ps1 and Edit-PersonaEngineConfig.ps1
- Six docs, two pipelines, traceability matrix, V-5a and sanitization records
Three deviations from tasks.md, each recorded in its status block
T033 is not in Resolve-UserPersona. evaluationErrorThreshold is run-level
state and the rule engine is pure; a counter there would break Principle IV.
It lives in New-PersonaRunCounter and is applied in the run loop.
A new src/Engine/ layer holds Invoke-PersonaEngineRun. The entry script
imports the manifest, which requires Microsoft.Graph.Authentication, so a
loop living only inside it could not run on a machine without the Graph SDK
and SC-004 could not be proven at all. The entry script is now a thin
wrapper and what ships is what is tested.
The invalid-config corpus is generated by a committed script, with the
generated fixtures committed too, so a reviewer sees the fixture in the diff.
Defects found by running the code, not by reading it
Group and role ID lists were double-wrapped: @(Get-PersonaGroupIdPage ...)
around a comma-returned array collapsed every membership list into one
bogus space-joined entry. That is a silent false non-match, exactly what
FR-013 exists to prevent.
A 403 whose status appears only in the exception message parsed as $null,
which the retry policy treats as a transport error - five requests per
account against a tenant already refusing. Status extraction now falls back
to the message text, bounded to 400-599.
The sanitization scan walked tracked files only, so it covered 34 of 156
files and none of this phase's code. It now scans untracked non-ignored
files too, and a negative control confirms it catches a planted leak.
Test-Json reports one error per violating location, not first-failure-only
as the V-5a draft claimed. Record and pin corrected.
Enforcement remains blocked on the V-4 security sign-off.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,156 @@
|
||||
#Requires -Modules @{ ModuleName = 'Pester'; ModuleVersion = '5.0.0' }
|
||||
|
||||
<#
|
||||
FR-021, SC-007: Processed = Matched + Unclassified + EvaluationError at every
|
||||
summary, and a mismatch is reported as an engine defect.
|
||||
|
||||
The forced-mismatch tests are the point. A reconciliation check that only ever
|
||||
sees correct data proves that the arithmetic works, not that the failure path
|
||||
does - and the failure path is the only part anyone will ever depend on.
|
||||
#>
|
||||
|
||||
BeforeAll {
|
||||
$repoRoot = Split-Path (Split-Path $PSScriptRoot -Parent) -Parent
|
||||
. (Join-Path $repoRoot 'tests/TestHelpers.ps1')
|
||||
foreach ($file in (Get-PersonaSourceFile -RepoRoot $repoRoot)) { . $file }
|
||||
|
||||
$script:target = 'extension_<EXTENSION-APP-ID>_<PERSONA>'
|
||||
$script:population = @(New-TestPopulation -Count 15 -TargetAttribute $target)
|
||||
}
|
||||
|
||||
Describe 'Reconciliation arithmetic (FR-021)' {
|
||||
|
||||
It 'passes when the outcome buckets account for every processed user' {
|
||||
$counters = New-PersonaRunCounter -Rules @()
|
||||
$counters.Processed = 10
|
||||
$counters.Matched = 7
|
||||
$counters.Unclassified = 2
|
||||
$counters.EvaluationError = 1
|
||||
|
||||
Test-PersonaReconciliation -Counters $counters | Should -BeTrue
|
||||
}
|
||||
|
||||
It 'fails when a user was processed but landed in no bucket' {
|
||||
$counters = New-PersonaRunCounter -Rules @()
|
||||
$counters.Processed = 10
|
||||
$counters.Matched = 7
|
||||
$counters.Unclassified = 2
|
||||
$counters.EvaluationError = 0
|
||||
|
||||
Test-PersonaReconciliation -Counters $counters | Should -BeFalse
|
||||
}
|
||||
|
||||
It 'fails when a user was double-counted' {
|
||||
$counters = New-PersonaRunCounter -Rules @()
|
||||
$counters.Processed = 10
|
||||
$counters.Matched = 8
|
||||
$counters.Unclassified = 2
|
||||
$counters.EvaluationError = 1
|
||||
|
||||
Test-PersonaReconciliation -Counters $counters | Should -BeFalse
|
||||
}
|
||||
|
||||
It 'passes trivially on an empty run' {
|
||||
Test-PersonaReconciliation -Counters (New-PersonaRunCounter -Rules @()) | Should -BeTrue
|
||||
}
|
||||
|
||||
It 'ignores the action buckets, which would otherwise mask a lost user' {
|
||||
# Skipped is a catch-all. If reconciliation checked the action buckets, a lost
|
||||
# user absorbed into Skipped would keep the sums balanced on a broken run.
|
||||
$counters = New-PersonaRunCounter -Rules @()
|
||||
$counters.Processed = 5
|
||||
$counters.Matched = 5
|
||||
$counters.Skipped = 99
|
||||
|
||||
Test-PersonaReconciliation -Counters $counters | Should -BeTrue
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Reconciliation failure detail' {
|
||||
|
||||
It 'reports the difference and its sign so a maintainer can tell loss from duplication' {
|
||||
$counters = New-PersonaRunCounter -Rules @()
|
||||
$counters.Processed = 10
|
||||
$counters.Matched = 7
|
||||
$counters.Unclassified = 2
|
||||
$counters.EvaluationError = 0
|
||||
|
||||
$detail = Get-PersonaReconciliationDetail -Counters $counters
|
||||
|
||||
$detail.processed | Should -Be 10
|
||||
$detail.outcomeTotal | Should -Be 9
|
||||
$detail.difference | Should -Be 1
|
||||
$detail.severity | Should -Be 'Error'
|
||||
$detail.defect | Should -Be 'ReconciliationFailure'
|
||||
}
|
||||
|
||||
It 'reports a negative difference when a user was double-counted' {
|
||||
$counters = New-PersonaRunCounter -Rules @()
|
||||
$counters.Processed = 10
|
||||
$counters.Matched = 9
|
||||
$counters.Unclassified = 2
|
||||
|
||||
(Get-PersonaReconciliationDetail -Counters $counters).difference | Should -Be -1
|
||||
}
|
||||
}
|
||||
|
||||
Describe 'Reconciliation over a real run (SC-007)' {
|
||||
|
||||
BeforeEach {
|
||||
Mock Get-PersonaUsers { $script:population }
|
||||
Mock Write-Host { }
|
||||
}
|
||||
|
||||
It 'reconciles at every interim summary and at completion' {
|
||||
$config = New-TestRuntimeConfiguration -TargetAttribute $target -SummaryInterval 5
|
||||
|
||||
$info = $null
|
||||
$null = Invoke-PersonaEngineRun -Configuration $config -TargetAttribute $target `
|
||||
-Context (New-TestAuditContext) -AuditParameters @{ Destination = 'stream' } `
|
||||
-InformationVariable info
|
||||
|
||||
$summaries = Get-CapturedAuditRecord -Captured $info -RecordType 'Summary'
|
||||
|
||||
# Three interim boundaries at 5, 10, 15 plus the final summary.
|
||||
$summaries.Count | Should -BeGreaterThan 1
|
||||
foreach ($summary in $summaries) {
|
||||
$summary['reconciliationPassed'] | Should -BeTrue
|
||||
$summary['processed'] | Should -Be ($summary['matched'] + $summary['unclassified'] + $summary['evaluationError'])
|
||||
}
|
||||
}
|
||||
|
||||
It 'returns exit code 5 and emits an EngineDefect when reconciliation fails' {
|
||||
# The failure is forced by making Add-PersonaRunResult drop the outcome tally
|
||||
# while still counting the user as processed - precisely the bookkeeping defect
|
||||
# FR-021 exists to catch.
|
||||
Mock Add-PersonaRunResult { $Counters.Processed++ }
|
||||
|
||||
$config = New-TestRuntimeConfiguration -TargetAttribute $target -SummaryInterval 0
|
||||
|
||||
$info = $null
|
||||
$outcome = Invoke-PersonaEngineRun -Configuration $config -TargetAttribute $target `
|
||||
-Context (New-TestAuditContext) -AuditParameters @{ Destination = 'stream' } `
|
||||
-InformationVariable info
|
||||
|
||||
$outcome.ExitCode | Should -Be 5
|
||||
|
||||
$defects = Get-CapturedAuditRecord -Captured $info -RecordType 'EngineDefect'
|
||||
|
||||
$defects.Count | Should -BeGreaterThan 0
|
||||
$defects[0]['defect'] | Should -Be 'ReconciliationFailure'
|
||||
$defects[0]['severity'] | Should -Be 'Error'
|
||||
}
|
||||
|
||||
It 'marks the summary record itself as failed when reconciliation fails' {
|
||||
Mock Add-PersonaRunResult { $Counters.Processed++ }
|
||||
|
||||
$config = New-TestRuntimeConfiguration -TargetAttribute $target -SummaryInterval 0
|
||||
|
||||
$info = $null
|
||||
$null = Invoke-PersonaEngineRun -Configuration $config -TargetAttribute $target `
|
||||
-Context (New-TestAuditContext) -AuditParameters @{ Destination = 'stream' } `
|
||||
-InformationVariable info
|
||||
|
||||
(Get-CapturedAuditRecord -Captured $info -RecordType 'Summary')[0]['reconciliationPassed'] | Should -BeFalse
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user