Files
personaEngine2/src/Configuration/Test-PersonaConfiguration.ps1
T
dave cdc6bb33d3 Implement Stage A: rule engine, validation, audit, and safety gates
Completes 109 of 121 tasks. Every remaining task needs a tenant connection
(T055, T056, T101-T103) or an Azure Automation account (T115-T121).

  354 offline Pester tests      PASS
  Engine purity (Principle IV)  PASS
  Sanitization (SC-013)         PASS  (156 files)
  Graph module loaded in tests  none  (SC-008 holds)

What landed
  - Four-layer configuration validation with stable finding codes, covering
    every VR-002 and VR-003 condition, plus a 23-fixture invalid-config corpus
  - Run loop, audit records (NDJSON through a single sink), summaries,
    reconciliation, and exit codes 0-6
  - Persistence behind a single write-body builder whose result always has
    exactly one key
  - Invoke-PersonaEngine.ps1 and Edit-PersonaEngineConfig.ps1
  - Six docs, two pipelines, traceability matrix, V-5a and sanitization records

Three deviations from tasks.md, each recorded in its status block

  T033 is not in Resolve-UserPersona. evaluationErrorThreshold is run-level
  state and the rule engine is pure; a counter there would break Principle IV.
  It lives in New-PersonaRunCounter and is applied in the run loop.

  A new src/Engine/ layer holds Invoke-PersonaEngineRun. The entry script
  imports the manifest, which requires Microsoft.Graph.Authentication, so a
  loop living only inside it could not run on a machine without the Graph SDK
  and SC-004 could not be proven at all. The entry script is now a thin
  wrapper and what ships is what is tested.

  The invalid-config corpus is generated by a committed script, with the
  generated fixtures committed too, so a reviewer sees the fixture in the diff.

Defects found by running the code, not by reading it

  Group and role ID lists were double-wrapped: @(Get-PersonaGroupIdPage ...)
  around a comma-returned array collapsed every membership list into one
  bogus space-joined entry. That is a silent false non-match, exactly what
  FR-013 exists to prevent.

  A 403 whose status appears only in the exception message parsed as $null,
  which the retry policy treats as a transport error - five requests per
  account against a tenant already refusing. Status extraction now falls back
  to the message text, bounded to 400-599.

  The sanitization scan walked tracked files only, so it covered 34 of 156
  files and none of this phase's code. It now scans untracked non-ignored
  files too, and a negative control confirms it catches a planted leak.

  Test-Json reports one error per violating location, not first-failure-only
  as the V-5a draft claimed. Record and pin corrected.

Enforcement remains blocked on the V-4 security sign-off.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 21:48:19 -04:00

262 lines
11 KiB
PowerShell

function Test-PersonaConfiguration {
<#
.SYNOPSIS
Runs all four validation layers over a configuration file (VR-001, FR-002).
.DESCRIPTION
Layers run in order and fail fast between them:
1. Syntax ConvertFrom-Json PE-SYN-nnn
2. Schema Test-Json -SchemaFile PE-SCH-nnn
3. Semantic Test-PersonaConfigurationSemantic PE-SEM-nnn
4. Safety Test-PersonaConfigurationSafety PE-SAF-nnn
A layer that produces Error findings stops the sequence. Running semantic
checks over a structurally invalid document produces noise, not signal: every
missing field yields a cascade of consequent errors, and the author has to
guess which one is the actual cause.
Layer 2 error handling is driven by the V-5a observation
(specs/001-persona-engine/verification/V-5a.md), which matters more than it
looks. On PowerShell 7.6.5, Test-Json returns $true when the SCHEMA ITSELF is
unparseable, writing the failure to the error stream instead. A wrapper that
trusted the return value would report a configuration as schema-valid when
the schema never ran. So this function treats a non-empty error variable as
failure regardless of what was returned, and separates the "schema is broken"
case from the "configuration is invalid" case, because they need different
exit codes.
.PARAMETER Path
Configuration file to validate.
.PARAMETER SchemaPath
Schema override. Defaults to the shipped config/persona-engine.schema.json.
.PARAMETER PreviousConfigPath
A previously deployed configuration, enabling the VR-003 comparison checks
(version downgrade, undeclared rule deletion or reorder).
.PARAMETER EnforcementEnabled
Whether this configuration will be used for an enforcing run. Raises the
severity of several safety findings.
.PARAMETER SkipSafety
Runs layers 1-3 only. Used by the editor while a document is mid-edit.
.OUTPUTS
PersonaEngine.ValidationResult
#>
[CmdletBinding()]
[OutputType([pscustomobject])]
param(
[Parameter(Mandatory)]
[ValidateNotNullOrEmpty()]
[string] $Path,
[string] $SchemaPath,
[string] $PreviousConfigPath,
[switch] $EnforcementEnabled,
[switch] $SkipSafety
)
$findings = [System.Collections.Generic.List[object]]::new()
$document = $null
$stoppedAtLayer = $null
$schemaUnusable = $false
# ---------------------------------------------------------------- Layer 1
if (-not (Test-Path -LiteralPath $Path -PathType Leaf)) {
$findings.Add((New-PersonaValidationFinding -Severity Error -Code 'PE-SYN-001' `
-Location $Path `
-Description "Configuration file not found or is not a file: '$Path'." `
-SuggestedResolution 'Check the path. In a pipeline, confirm the file was checked out and the working directory is what you expect.'))
return New-PersonaValidationResult -Findings $findings -Document $null -StoppedAtLayer 'Syntax' -SchemaUnusable $false
}
$raw = $null
try {
$raw = Get-Content -LiteralPath $Path -Raw -ErrorAction Stop
}
catch {
$findings.Add((New-PersonaValidationFinding -Severity Error -Code 'PE-SYN-002' `
-Location $Path `
-Description "Configuration file could not be read: $($_.Exception.Message)" `
-SuggestedResolution 'Check file permissions and that no other process holds an exclusive lock.'))
return New-PersonaValidationResult -Findings $findings -Document $null -StoppedAtLayer 'Syntax' -SchemaUnusable $false
}
try {
$document = $raw | ConvertFrom-Json -Depth 32 -ErrorAction Stop
}
catch {
$findings.Add((New-PersonaValidationFinding -Severity Error -Code 'PE-SYN-003' `
-Location $Path `
-Description "Configuration is not valid JSON: $($_.Exception.Message)" `
-SuggestedResolution 'Fix the JSON syntax. A trailing comma or an unquoted key is the usual cause.'))
return New-PersonaValidationResult -Findings $findings -Document $null -StoppedAtLayer 'Syntax' -SchemaUnusable $false
}
# ---------------------------------------------------------------- Layer 2
if (-not $SchemaPath) {
$moduleRoot = Split-Path (Split-Path $PSScriptRoot -Parent) -Parent
$SchemaPath = Join-Path $moduleRoot 'config/persona-engine.schema.json'
}
if (-not (Test-Path -LiteralPath $SchemaPath -PathType Leaf)) {
$findings.Add((New-PersonaValidationFinding -Severity Error -Code 'PE-SCH-002' `
-Location $SchemaPath `
-Description "Schema file not found: '$SchemaPath'." `
-SuggestedResolution 'Supply -SchemaPath, or restore config/persona-engine.schema.json.'))
return New-PersonaValidationResult -Findings $findings -Document $document -StoppedAtLayer 'Schema' -SchemaUnusable $true
}
$schemaErrors = $null
$schemaOk = $raw | Test-Json -SchemaFile $SchemaPath -ErrorAction SilentlyContinue -ErrorVariable schemaErrors
foreach ($schemaError in @($schemaErrors)) {
$message = [string]$schemaError.Exception.Message
# V-5a: this message arrives with a $true return value. Treating it as a
# pass would validate every configuration against a schema that never ran.
if ($message -match 'Cannot parse the JSON schema') {
$schemaUnusable = $true
$findings.Add((New-PersonaValidationFinding -Severity Error -Code 'PE-SCH-003' `
-Location $SchemaPath `
-Description "The schema file itself is not valid JSON Schema and could not be used: $message" `
-SuggestedResolution 'Repair the schema file. Until it parses, no configuration can be schema-validated, and a passing result would be meaningless.'))
continue
}
$findings.Add((New-PersonaValidationFinding -Severity Error -Code 'PE-SCH-001' `
-Location (Get-PersonaSchemaErrorLocation -Message $message) `
-Description $message `
-SuggestedResolution 'Correct the document to match config/persona-engine.schema.json. Test-Json reports only the first violation per run, so re-validate after each fix.'))
}
if ($schemaUnusable -or -not $schemaOk -or @($schemaErrors).Count -gt 0) {
return New-PersonaValidationResult -Findings $findings -Document $document -StoppedAtLayer 'Schema' -SchemaUnusable $schemaUnusable
}
# ---------------------------------------------------------------- Layer 3
foreach ($finding in (Test-PersonaConfigurationSemantic -Document $document)) { $findings.Add($finding) }
if (@($findings | Where-Object Severity -EQ 'Error').Count -gt 0) {
return New-PersonaValidationResult -Findings $findings -Document $document -StoppedAtLayer 'Semantic' -SchemaUnusable $false
}
# ---------------------------------------------------------------- Layer 4
if (-not $SkipSafety) {
$safetyParams = @{ Document = $document; EnforcementEnabled = $EnforcementEnabled }
if ($PreviousConfigPath) { $safetyParams['PreviousConfigPath'] = $PreviousConfigPath }
foreach ($finding in (Test-PersonaConfigurationSafety @safetyParams)) { $findings.Add($finding) }
}
New-PersonaValidationResult -Findings $findings -Document $document -StoppedAtLayer $null -SchemaUnusable $false
}
function New-PersonaValidationResult {
<#
.SYNOPSIS
Wraps a finding collection into the shape every caller consumes.
.DESCRIPTION
IsValid is computed here rather than by each caller, so console, pipeline,
and editor cannot disagree about what "valid" means. Warnings never affect
IsValid; escalation under -TreatWarningsAsErrors is the caller's decision
(VR-005) and belongs where the exit code is chosen.
#>
[CmdletBinding()]
[OutputType([pscustomobject])]
param(
[Parameter(Mandatory)] [AllowEmptyCollection()] [object] $Findings,
[AllowNull()] [object] $Document,
[AllowNull()] [string] $StoppedAtLayer,
[bool] $SchemaUnusable
)
$all = @($Findings)
[pscustomobject]@{
PSTypeName = 'PersonaEngine.ValidationResult'
IsValid = (@($all | Where-Object Severity -EQ 'Error').Count -eq 0)
Findings = $all
ErrorCount = @($all | Where-Object Severity -EQ 'Error').Count
WarningCount = @($all | Where-Object Severity -EQ 'Warning').Count
Document = $Document
StoppedAtLayer = $StoppedAtLayer
SchemaUnusable = $SchemaUnusable
}
}
function Get-PersonaSchemaErrorLocation {
<#
.SYNOPSIS
Extracts the JSON pointer from a Test-Json error message.
.DESCRIPTION
Test-Json embeds the failing location in prose - "... at '/rules/3/priority'".
VR-004 requires a location on every finding, so it is lifted out here rather
than leaving the caller to read it out of the description. When no pointer is
present the document root is reported, which is honest: the violation is
somewhere in the document and the message says where in words.
#>
[CmdletBinding()]
[OutputType([string])]
param([Parameter(Mandatory)] [AllowEmptyString()] [string] $Message)
if ($Message -match "at '([^']*)'") {
$pointer = $Matches[1]
return $pointer ? $pointer : '/'
}
'/'
}
function Write-PersonaValidationFinding {
<#
.SYNOPSIS
Renders validation findings for a human reader.
.DESCRIPTION
Grouped by severity, most serious first, with the suggested resolution on
its own line. A finding without a visible resolution just tells someone they
are wrong, which is why VR-004 makes the field mandatory and why it is
printed rather than hidden behind a verbose switch.
#>
[CmdletBinding()]
param(
[Parameter(Mandatory)] [AllowEmptyCollection()] [object[]] $Findings
)
if (@($Findings).Count -eq 0) {
Write-Host 'Validation passed with no findings.' -ForegroundColor Green
return
}
foreach ($severity in @('Error', 'Warning', 'Information')) {
$set = @($Findings | Where-Object Severity -EQ $severity)
if ($set.Count -eq 0) { continue }
$colour = switch ($severity) { 'Error' { 'Red' } 'Warning' { 'Yellow' } default { 'Gray' } }
foreach ($finding in $set) {
Write-Host ("[{0}] {1} {2}" -f $finding.Severity.ToUpperInvariant(), $finding.Code, $finding.Location) -ForegroundColor $colour
Write-Host (" {0}" -f $finding.Description)
Write-Host (" -> {0}" -f $finding.SuggestedResolution) -ForegroundColor DarkGray
}
}
}