Files
personaEngine2/pipelines/validate.yml
T
dave cdc6bb33d3 Implement Stage A: rule engine, validation, audit, and safety gates
Completes 109 of 121 tasks. Every remaining task needs a tenant connection
(T055, T056, T101-T103) or an Azure Automation account (T115-T121).

  354 offline Pester tests      PASS
  Engine purity (Principle IV)  PASS
  Sanitization (SC-013)         PASS  (156 files)
  Graph module loaded in tests  none  (SC-008 holds)

What landed
  - Four-layer configuration validation with stable finding codes, covering
    every VR-002 and VR-003 condition, plus a 23-fixture invalid-config corpus
  - Run loop, audit records (NDJSON through a single sink), summaries,
    reconciliation, and exit codes 0-6
  - Persistence behind a single write-body builder whose result always has
    exactly one key
  - Invoke-PersonaEngine.ps1 and Edit-PersonaEngineConfig.ps1
  - Six docs, two pipelines, traceability matrix, V-5a and sanitization records

Three deviations from tasks.md, each recorded in its status block

  T033 is not in Resolve-UserPersona. evaluationErrorThreshold is run-level
  state and the rule engine is pure; a counter there would break Principle IV.
  It lives in New-PersonaRunCounter and is applied in the run loop.

  A new src/Engine/ layer holds Invoke-PersonaEngineRun. The entry script
  imports the manifest, which requires Microsoft.Graph.Authentication, so a
  loop living only inside it could not run on a machine without the Graph SDK
  and SC-004 could not be proven at all. The entry script is now a thin
  wrapper and what ships is what is tested.

  The invalid-config corpus is generated by a committed script, with the
  generated fixtures committed too, so a reviewer sees the fixture in the diff.

Defects found by running the code, not by reading it

  Group and role ID lists were double-wrapped: @(Get-PersonaGroupIdPage ...)
  around a comma-returned array collapsed every membership list into one
  bogus space-joined entry. That is a silent false non-match, exactly what
  FR-013 exists to prevent.

  A 403 whose status appears only in the exception message parsed as $null,
  which the retry policy treats as a transport error - five requests per
  account against a tenant already refusing. Status extraction now falls back
  to the message text, bounded to 400-599.

  The sanitization scan walked tracked files only, so it covered 34 of 156
  files and none of this phase's code. It now scans untracked non-ignored
  files too, and a negative control confirms it catches a planted leak.

  Test-Json reports one error per violating location, not first-failure-only
  as the V-5a draft claimed. Record and pin corrected.

Enforcement remains blocked on the V-4 security sign-off.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 21:48:19 -04:00

176 lines
6.3 KiB
YAML

# Persona Engine - validation stage
#
# Everything here runs with no tenant, no credentials, and no network (SC-008). That
# is the point: a pipeline that needs a directory connection to tell you a rule file
# is wrong cannot run on every pull request, and the check that only runs sometimes is
# the one that stops catching things.
#
# Gate order is deliberate, cheapest and most categorical first. Sanitization runs
# before anything else because a leaked identifier in a branch is a problem whether or
# not the code compiles, and every later stage prints file contents into build logs.
trigger:
branches:
include:
- main
paths:
include:
- src/*
- tests/*
- config/*
- pipelines/*
- Invoke-PersonaEngine.ps1
- Edit-PersonaEngineConfig.ps1
- PersonaEngine.psd1
- PersonaEngine.psm1
pr:
branches:
include:
- main
pool:
vmImage: windows-latest
variables:
# Pinned rather than latest. A validation stage that changes behaviour when an
# upstream module publishes is not a gate, it is a coin flip.
pesterVersion: '5.6.1'
analyzerVersion: '1.22.0'
steps:
- checkout: self
fetchDepth: 0
- task: PowerShell@2
displayName: 'Gate 1 - Sanitization (SC-013)'
inputs:
pwsh: true
filePath: tests/Test-Sanitization.ps1
failOnStderr: false
- task: PowerShell@2
displayName: 'Install pinned analysis modules'
inputs:
pwsh: true
targetType: inline
script: |
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
Install-Module Pester -RequiredVersion $(pesterVersion) -Force -SkipPublisherCheck -Scope CurrentUser
Install-Module PSScriptAnalyzer -RequiredVersion $(analyzerVersion) -Force -Scope CurrentUser
- task: PowerShell@2
displayName: 'Gate 2 - PSScriptAnalyzer'
inputs:
pwsh: true
targetType: inline
script: |
$ErrorActionPreference = 'Stop'
$findings = Invoke-ScriptAnalyzer -Path . -Recurse -Settings ./PSScriptAnalyzerSettings.psd1
if ($findings) {
$findings | Format-Table -AutoSize | Out-String | Write-Host
}
$blocking = @($findings | Where-Object Severity -in 'Error', 'Warning')
if ($blocking.Count -gt 0) {
throw "PSScriptAnalyzer reported $($blocking.Count) blocking finding(s)."
}
- task: PowerShell@2
displayName: 'Gate 3 - Engine purity (Principle IV)'
inputs:
pwsh: true
filePath: tests/Test-EnginePurity.ps1
- task: PowerShell@2
displayName: 'Gate 4 - Shipped schema is valid JSON Schema'
inputs:
pwsh: true
targetType: inline
script: |
$ErrorActionPreference = 'Stop'
# V-5a: Test-Json returns $true when the schema itself cannot be parsed, so a
# broken schema would let every later check pass while validating nothing.
# This stage exists solely to catch that.
$errors = $null
$null = '{}' | Test-Json -SchemaFile ./config/persona-engine.schema.json -ErrorAction SilentlyContinue -ErrorVariable errors
$unusable = @($errors | Where-Object { $_.Exception.Message -match 'Cannot parse the JSON schema' })
if ($unusable.Count -gt 0) {
throw 'config/persona-engine.schema.json is not valid JSON Schema. No configuration can be schema-validated until it is repaired.'
}
# The contract copy and the shipped copy must stay identical, or a rule author
# reading the contract validates against a different schema than the engine.
$shipped = (Get-FileHash ./config/persona-engine.schema.json -Algorithm SHA256).Hash
$contract = (Get-FileHash ./specs/001-persona-engine/contracts/persona-engine.schema.json -Algorithm SHA256).Hash
if ($shipped -ne $contract) {
throw 'config/persona-engine.schema.json and the contract copy have diverged.'
}
- task: PowerShell@2
displayName: 'Gate 5 - Example configuration passes all four layers'
inputs:
pwsh: true
targetType: inline
script: |
$ErrorActionPreference = 'Stop'
./Edit-PersonaEngineConfig.ps1 -ConfigPath ./config/persona-engine.example.json -ValidateOnly -NonInteractive
if ($LASTEXITCODE -ne 0) { throw "The shipped example configuration failed validation with exit code $LASTEXITCODE." }
- task: PowerShell@2
displayName: 'Gate 6 - Offline Pester suite (SC-008)'
inputs:
pwsh: true
targetType: inline
script: |
$ErrorActionPreference = 'Stop'
$config = & ./tests/PesterConfiguration.ps1 -Suite Offline
$config.Run.Exit = $false
$config.Run.PassThru = $true
$config.TestResult.Enabled = $true
$config.TestResult.OutputPath = './testResults.offline.xml'
$config.Output.Verbosity = 'Normal'
$result = Invoke-Pester -Configuration $config
# Asserted, not assumed. A configuration change that silently filtered every
# test out would otherwise report a green pipeline over zero coverage.
if ($result.TotalCount -eq 0) { throw 'The offline suite ran no tests.' }
if ($result.FailedCount -gt 0) { throw "$($result.FailedCount) offline test(s) failed." }
Write-Host "Offline suite: $($result.PassedCount) passed, $($result.FailedCount) failed."
- task: PublishTestResults@2
displayName: 'Publish offline test results'
condition: succeededOrFailed()
inputs:
testResultsFormat: NUnit
testResultsFiles: './testResults.offline.xml'
testRunTitle: 'Persona Engine - offline suite'
- task: PowerShell@2
displayName: 'Gate 7 - No Graph module was loaded (SC-008)'
inputs:
pwsh: true
targetType: inline
script: |
# The proof that the offline suite is genuinely offline. If a test ever
# imports the Graph SDK, the "runs with no tenant" claim quietly stops being
# true and nobody notices until an air-gapped build fails.
$config = & ./tests/PesterConfiguration.ps1 -Suite Offline
$config.Run.PassThru = $true
$config.Output.Verbosity = 'None'
$null = Invoke-Pester -Configuration $config
$graph = Get-Module | Where-Object Name -like 'Microsoft.Graph*'
if ($graph) {
throw "A Graph module was loaded during the offline suite: $($graph.Name -join ', ')"
}
Write-Host 'No Graph module was loaded. SC-008 holds.'