Files
personaEngine2/pipelines/test.yml
T
dave cdc6bb33d3 Implement Stage A: rule engine, validation, audit, and safety gates
Completes 109 of 121 tasks. Every remaining task needs a tenant connection
(T055, T056, T101-T103) or an Azure Automation account (T115-T121).

  354 offline Pester tests      PASS
  Engine purity (Principle IV)  PASS
  Sanitization (SC-013)         PASS  (156 files)
  Graph module loaded in tests  none  (SC-008 holds)

What landed
  - Four-layer configuration validation with stable finding codes, covering
    every VR-002 and VR-003 condition, plus a 23-fixture invalid-config corpus
  - Run loop, audit records (NDJSON through a single sink), summaries,
    reconciliation, and exit codes 0-6
  - Persistence behind a single write-body builder whose result always has
    exactly one key
  - Invoke-PersonaEngine.ps1 and Edit-PersonaEngineConfig.ps1
  - Six docs, two pipelines, traceability matrix, V-5a and sanitization records

Three deviations from tasks.md, each recorded in its status block

  T033 is not in Resolve-UserPersona. evaluationErrorThreshold is run-level
  state and the rule engine is pure; a counter there would break Principle IV.
  It lives in New-PersonaRunCounter and is applied in the run loop.

  A new src/Engine/ layer holds Invoke-PersonaEngineRun. The entry script
  imports the manifest, which requires Microsoft.Graph.Authentication, so a
  loop living only inside it could not run on a machine without the Graph SDK
  and SC-004 could not be proven at all. The entry script is now a thin
  wrapper and what ships is what is tested.

  The invalid-config corpus is generated by a committed script, with the
  generated fixtures committed too, so a reviewer sees the fixture in the diff.

Defects found by running the code, not by reading it

  Group and role ID lists were double-wrapped: @(Get-PersonaGroupIdPage ...)
  around a comma-returned array collapsed every membership list into one
  bogus space-joined entry. That is a silent false non-match, exactly what
  FR-013 exists to prevent.

  A 403 whose status appears only in the exception message parsed as $null,
  which the retry policy treats as a transport error - five requests per
  account against a tenant already refusing. Status extraction now falls back
  to the message text, bounded to 400-599.

  The sanitization scan walked tracked files only, so it covered 34 of 156
  files and none of this phase's code. It now scans untracked non-ignored
  files too, and a negative control confirms it catches a planted leak.

  Test-Json reports one error per violating location, not first-failure-only
  as the V-5a draft claimed. Record and pin corrected.

Enforcement remains blocked on the V-4 security sign-off.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 21:48:19 -04:00

182 lines
6.8 KiB
YAML

# Persona Engine - test stage
#
# Separate from validate.yml because the two answer different questions. Validation
# asks "is this repository well-formed?" and gates every pull request. This asks "does
# the engine behave correctly?" and publishes evidence.
#
# The Integration suite is present but disabled by default. It needs a delegated
# read-only connection to a real tenant (Stage A2), which a shared build agent cannot
# hold without storing credentials - and NFR-006 puts no secrets in source control.
# Enable it only on an agent with an interactive or workload-identity connection,
# never by adding a secret to this file.
trigger: none
pr: none
schedules:
- cron: '0 6 * * 1-5'
displayName: 'Weekday morning regression'
branches:
include:
- main
always: false
pool:
vmImage: windows-latest
parameters:
- name: runIntegration
displayName: 'Run the Integration suite (requires a tenant connection)'
type: boolean
default: false
variables:
pesterVersion: '5.6.1'
stages:
- stage: OfflineSuites
displayName: 'Offline and safety suites'
jobs:
- job: Offline
displayName: 'Offline suite'
steps:
- checkout: self
- task: PowerShell@2
displayName: 'Install pinned Pester'
inputs:
pwsh: true
targetType: inline
script: |
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
Install-Module Pester -RequiredVersion $(pesterVersion) -Force -SkipPublisherCheck -Scope CurrentUser
- task: PowerShell@2
displayName: 'Offline suite'
inputs:
pwsh: true
targetType: inline
script: |
$ErrorActionPreference = 'Stop'
$config = & ./tests/PesterConfiguration.ps1 -Suite Offline
$config.Run.Exit = $false
$config.Run.PassThru = $true
$config.TestResult.Enabled = $true
$config.TestResult.OutputPath = './testResults.offline.xml'
$config.CodeCoverage.Enabled = $true
$config.CodeCoverage.Path = @('./src')
$config.CodeCoverage.OutputPath = './coverage.offline.xml'
$config.Output.Verbosity = 'Normal'
$result = Invoke-Pester -Configuration $config
if ($result.TotalCount -eq 0) { throw 'The offline suite ran no tests.' }
if ($result.FailedCount -gt 0) { throw "$($result.FailedCount) offline test(s) failed." }
- task: PublishTestResults@2
displayName: 'Publish offline results'
condition: succeededOrFailed()
inputs:
testResultsFormat: NUnit
testResultsFiles: './testResults.offline.xml'
testRunTitle: 'Persona Engine - offline suite'
- task: PublishCodeCoverageResults@2
displayName: 'Publish coverage'
condition: succeededOrFailed()
inputs:
summaryFileLocation: './coverage.offline.xml'
- job: Safety
displayName: 'Safety suite'
dependsOn: Offline
steps:
- checkout: self
- task: PowerShell@2
displayName: 'Install pinned Pester'
inputs:
pwsh: true
targetType: inline
script: |
Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
Install-Module Pester -RequiredVersion $(pesterVersion) -Force -SkipPublisherCheck -Scope CurrentUser
- task: PowerShell@2
displayName: 'Safety suite (SC-002, SC-004, SC-005)'
inputs:
pwsh: true
targetType: inline
script: |
$ErrorActionPreference = 'Stop'
$config = & ./tests/PesterConfiguration.ps1 -Suite Safety
$config.Run.Exit = $false
$config.Run.PassThru = $true
$config.TestResult.Enabled = $true
$config.TestResult.OutputPath = './testResults.safety.xml'
$config.Output.Verbosity = 'Detailed'
$result = Invoke-Pester -Configuration $config
# The safety suite running zero tests is the most dangerous possible
# green build: it is exactly what a mis-tagged file looks like, and the
# assertions it drops are the zero-write and single-attribute ones.
if ($result.TotalCount -eq 0) { throw 'The safety suite ran no tests. Check the Safety tag.' }
if ($result.FailedCount -gt 0) { throw "$($result.FailedCount) safety test(s) failed. Do not deploy." }
Write-Host "Safety suite: $($result.PassedCount) passed."
- task: PublishTestResults@2
displayName: 'Publish safety results'
condition: succeededOrFailed()
inputs:
testResultsFormat: NUnit
testResultsFiles: './testResults.safety.xml'
testRunTitle: 'Persona Engine - safety suite'
- stage: IntegrationSuite
displayName: 'Integration suite (tenant required)'
dependsOn: OfflineSuites
condition: and(succeeded(), eq('${{ parameters.runIntegration }}', true))
jobs:
- job: Integration
displayName: 'Integration suite'
steps:
- checkout: self
- task: PowerShell@2
displayName: 'Integration suite'
inputs:
pwsh: true
targetType: inline
script: |
$ErrorActionPreference = 'Stop'
# No credential handling here by design. The agent must already hold a
# delegated read-only connection; if it does not, this fails loudly
# rather than prompting or falling back to a stored secret.
if (-not (Get-Module -ListAvailable Microsoft.Graph.Authentication)) {
throw 'Microsoft.Graph.Authentication is not available on this agent.'
}
$config = & ./tests/PesterConfiguration.ps1 -Suite Integration
$config.Run.Exit = $false
$config.Run.PassThru = $true
$config.TestResult.Enabled = $true
$config.TestResult.OutputPath = './testResults.integration.xml'
$result = Invoke-Pester -Configuration $config
if ($result.FailedCount -gt 0) { throw "$($result.FailedCount) integration test(s) failed." }
- task: PublishTestResults@2
displayName: 'Publish integration results'
condition: succeededOrFailed()
inputs:
testResultsFormat: NUnit
testResultsFiles: './testResults.integration.xml'
testRunTitle: 'Persona Engine - integration suite'