Implement Stage A: rule engine, validation, audit, and safety gates

Completes 109 of 121 tasks. Every remaining task needs a tenant connection (T055, T056, T101-T103) or an Azure Automation account (T115-T121). 354 offline Pester tests PASS Engine purity (Principle IV) PASS Sanitization (SC-013) PASS (156 files) Graph module loaded in tests none (SC-008 holds) What landed - Four-layer configuration validation with stable finding codes, covering every VR-002 and VR-003 condition, plus a 23-fixture invalid-config corpus - Run loop, audit records (NDJSON through a single sink), summaries, reconciliation, and exit codes 0-6 - Persistence behind a single write-body builder whose result always has exactly one key - Invoke-PersonaEngine.ps1 and Edit-PersonaEngineConfig.ps1 - Six docs, two pipelines, traceability matrix, V-5a and sanitization records Three deviations from tasks.md, each recorded in its status block T033 is not in Resolve-UserPersona. evaluationErrorThreshold is run-level state and the rule engine is pure; a counter there would break Principle IV. It lives in New-PersonaRunCounter and is applied in the run loop. A new src/Engine/ layer holds Invoke-PersonaEngineRun. The entry script imports the manifest, which requires Microsoft.Graph.Authentication, so a loop living only inside it could not run on a machine without the Graph SDK and SC-004 could not be proven at all. The entry script is now a thin wrapper and what ships is what is tested. The invalid-config corpus is generated by a committed script, with the generated fixtures committed too, so a reviewer sees the fixture in the diff. Defects found by running the code, not by reading it Group and role ID lists were double-wrapped: @(Get-PersonaGroupIdPage ...) around a comma-returned array collapsed every membership list into one bogus space-joined entry. That is a silent false non-match, exactly what FR-013 exists to prevent. A 403 whose status appears only in the exception message parsed as $null, which the retry policy treats as a transport error - five requests per account against a tenant already refusing. Status extraction now falls back to the message text, bounded to 400-599. The sanitization scan walked tracked files only, so it covered 34 of 156 files and none of this phase's code. It now scans untracked non-ignored files too, and a negative control confirms it catches a planted leak. Test-Json reports one error per violating location, not first-failure-only as the V-5a draft claimed. Record and pin corrected. Enforcement remains blocked on the V-4 security sign-off. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 21:48:19 -04:00
parent c59c85dd55
commit cdc6bb33d3
124 changed files with 16638 additions and 199 deletions
@@ -0,0 +1,181 @@
+# Persona Engine - test stage
+#
+# Separate from validate.yml because the two answer different questions. Validation
+# asks "is this repository well-formed?" and gates every pull request. This asks "does
+# the engine behave correctly?" and publishes evidence.
+#
+# The Integration suite is present but disabled by default. It needs a delegated
+# read-only connection to a real tenant (Stage A2), which a shared build agent cannot
+# hold without storing credentials - and NFR-006 puts no secrets in source control.
+# Enable it only on an agent with an interactive or workload-identity connection,
+# never by adding a secret to this file.
+
+trigger: none
+
+pr: none
+
+schedules:
+  - cron: '0 6 * * 1-5'
+    displayName: 'Weekday morning regression'
+    branches:
+      include:
+        - main
+    always: false
+
+pool:
+  vmImage: windows-latest
+
+parameters:
+  - name: runIntegration
+    displayName: 'Run the Integration suite (requires a tenant connection)'
+    type: boolean
+    default: false
+
+variables:
+  pesterVersion: '5.6.1'
+
+stages:
+  - stage: OfflineSuites
+    displayName: 'Offline and safety suites'
+    jobs:
+      - job: Offline
+        displayName: 'Offline suite'
+        steps:
+          - checkout: self
+
+          - task: PowerShell@2
+            displayName: 'Install pinned Pester'
+            inputs:
+              pwsh: true
+              targetType: inline
+              script: |
+                Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
+                Install-Module Pester -RequiredVersion $(pesterVersion) -Force -SkipPublisherCheck -Scope CurrentUser
+
+          - task: PowerShell@2
+            displayName: 'Offline suite'
+            inputs:
+              pwsh: true
+              targetType: inline
+              script: |
+                $ErrorActionPreference = 'Stop'
+
+                $config = & ./tests/PesterConfiguration.ps1 -Suite Offline
+                $config.Run.Exit = $false
+                $config.Run.PassThru = $true
+                $config.TestResult.Enabled = $true
+                $config.TestResult.OutputPath = './testResults.offline.xml'
+                $config.CodeCoverage.Enabled = $true
+                $config.CodeCoverage.Path = @('./src')
+                $config.CodeCoverage.OutputPath = './coverage.offline.xml'
+                $config.Output.Verbosity = 'Normal'
+
+                $result = Invoke-Pester -Configuration $config
+
+                if ($result.TotalCount -eq 0) { throw 'The offline suite ran no tests.' }
+                if ($result.FailedCount -gt 0) { throw "$($result.FailedCount) offline test(s) failed." }
+
+          - task: PublishTestResults@2
+            displayName: 'Publish offline results'
+            condition: succeededOrFailed()
+            inputs:
+              testResultsFormat: NUnit
+              testResultsFiles: './testResults.offline.xml'
+              testRunTitle: 'Persona Engine - offline suite'
+
+          - task: PublishCodeCoverageResults@2
+            displayName: 'Publish coverage'
+            condition: succeededOrFailed()
+            inputs:
+              summaryFileLocation: './coverage.offline.xml'
+
+      - job: Safety
+        displayName: 'Safety suite'
+        dependsOn: Offline
+        steps:
+          - checkout: self
+
+          - task: PowerShell@2
+            displayName: 'Install pinned Pester'
+            inputs:
+              pwsh: true
+              targetType: inline
+              script: |
+                Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
+                Install-Module Pester -RequiredVersion $(pesterVersion) -Force -SkipPublisherCheck -Scope CurrentUser
+
+          - task: PowerShell@2
+            displayName: 'Safety suite (SC-002, SC-004, SC-005)'
+            inputs:
+              pwsh: true
+              targetType: inline
+              script: |
+                $ErrorActionPreference = 'Stop'
+
+                $config = & ./tests/PesterConfiguration.ps1 -Suite Safety
+                $config.Run.Exit = $false
+                $config.Run.PassThru = $true
+                $config.TestResult.Enabled = $true
+                $config.TestResult.OutputPath = './testResults.safety.xml'
+                $config.Output.Verbosity = 'Detailed'
+
+                $result = Invoke-Pester -Configuration $config
+
+                # The safety suite running zero tests is the most dangerous possible
+                # green build: it is exactly what a mis-tagged file looks like, and the
+                # assertions it drops are the zero-write and single-attribute ones.
+                if ($result.TotalCount -eq 0) { throw 'The safety suite ran no tests. Check the Safety tag.' }
+                if ($result.FailedCount -gt 0) { throw "$($result.FailedCount) safety test(s) failed. Do not deploy." }
+
+                Write-Host "Safety suite: $($result.PassedCount) passed."
+
+          - task: PublishTestResults@2
+            displayName: 'Publish safety results'
+            condition: succeededOrFailed()
+            inputs:
+              testResultsFormat: NUnit
+              testResultsFiles: './testResults.safety.xml'
+              testRunTitle: 'Persona Engine - safety suite'
+
+  - stage: IntegrationSuite
+    displayName: 'Integration suite (tenant required)'
+    dependsOn: OfflineSuites
+    condition: and(succeeded(), eq('${{ parameters.runIntegration }}', true))
+    jobs:
+      - job: Integration
+        displayName: 'Integration suite'
+        steps:
+          - checkout: self
+
+          - task: PowerShell@2
+            displayName: 'Integration suite'
+            inputs:
+              pwsh: true
+              targetType: inline
+              script: |
+                $ErrorActionPreference = 'Stop'
+
+                # No credential handling here by design. The agent must already hold a
+                # delegated read-only connection; if it does not, this fails loudly
+                # rather than prompting or falling back to a stored secret.
+                if (-not (Get-Module -ListAvailable Microsoft.Graph.Authentication)) {
+                  throw 'Microsoft.Graph.Authentication is not available on this agent.'
+                }
+
+                $config = & ./tests/PesterConfiguration.ps1 -Suite Integration
+                $config.Run.Exit = $false
+                $config.Run.PassThru = $true
+                $config.TestResult.Enabled = $true
+                $config.TestResult.OutputPath = './testResults.integration.xml'
+
+                $result = Invoke-Pester -Configuration $config
+
+                if ($result.FailedCount -gt 0) { throw "$($result.FailedCount) integration test(s) failed." }
+
+          - task: PublishTestResults@2
+            displayName: 'Publish integration results'
+            condition: succeededOrFailed()
+            inputs:
+              testResultsFormat: NUnit
+              testResultsFiles: './testResults.integration.xml'
+              testRunTitle: 'Persona Engine - integration suite'
@@ -0,0 +1,175 @@
+# Persona Engine - validation stage
+#
+# Everything here runs with no tenant, no credentials, and no network (SC-008). That
+# is the point: a pipeline that needs a directory connection to tell you a rule file
+# is wrong cannot run on every pull request, and the check that only runs sometimes is
+# the one that stops catching things.
+#
+# Gate order is deliberate, cheapest and most categorical first. Sanitization runs
+# before anything else because a leaked identifier in a branch is a problem whether or
+# not the code compiles, and every later stage prints file contents into build logs.
+
+trigger:
+  branches:
+    include:
+      - main
+  paths:
+    include:
+      - src/*
+      - tests/*
+      - config/*
+      - pipelines/*
+      - Invoke-PersonaEngine.ps1
+      - Edit-PersonaEngineConfig.ps1
+      - PersonaEngine.psd1
+      - PersonaEngine.psm1
+
+pr:
+  branches:
+    include:
+      - main
+
+pool:
+  vmImage: windows-latest
+
+variables:
+  # Pinned rather than latest. A validation stage that changes behaviour when an
+  # upstream module publishes is not a gate, it is a coin flip.
+  pesterVersion: '5.6.1'
+  analyzerVersion: '1.22.0'
+
+steps:
+  - checkout: self
+    fetchDepth: 0
+
+  - task: PowerShell@2
+    displayName: 'Gate 1 - Sanitization (SC-013)'
+    inputs:
+      pwsh: true
+      filePath: tests/Test-Sanitization.ps1
+      failOnStderr: false
+
+  - task: PowerShell@2
+    displayName: 'Install pinned analysis modules'
+    inputs:
+      pwsh: true
+      targetType: inline
+      script: |
+        Set-PSRepository -Name PSGallery -InstallationPolicy Trusted
+        Install-Module Pester -RequiredVersion $(pesterVersion) -Force -SkipPublisherCheck -Scope CurrentUser
+        Install-Module PSScriptAnalyzer -RequiredVersion $(analyzerVersion) -Force -Scope CurrentUser
+
+  - task: PowerShell@2
+    displayName: 'Gate 2 - PSScriptAnalyzer'
+    inputs:
+      pwsh: true
+      targetType: inline
+      script: |
+        $ErrorActionPreference = 'Stop'
+        $findings = Invoke-ScriptAnalyzer -Path . -Recurse -Settings ./PSScriptAnalyzerSettings.psd1
+
+        if ($findings) {
+          $findings | Format-Table -AutoSize | Out-String | Write-Host
+        }
+
+        $blocking = @($findings | Where-Object Severity -in 'Error', 'Warning')
+        if ($blocking.Count -gt 0) {
+          throw "PSScriptAnalyzer reported $($blocking.Count) blocking finding(s)."
+        }
+
+  - task: PowerShell@2
+    displayName: 'Gate 3 - Engine purity (Principle IV)'
+    inputs:
+      pwsh: true
+      filePath: tests/Test-EnginePurity.ps1
+
+  - task: PowerShell@2
+    displayName: 'Gate 4 - Shipped schema is valid JSON Schema'
+    inputs:
+      pwsh: true
+      targetType: inline
+      script: |
+        $ErrorActionPreference = 'Stop'
+
+        # V-5a: Test-Json returns $true when the schema itself cannot be parsed, so a
+        # broken schema would let every later check pass while validating nothing.
+        # This stage exists solely to catch that.
+        $errors = $null
+        $null = '{}' | Test-Json -SchemaFile ./config/persona-engine.schema.json -ErrorAction SilentlyContinue -ErrorVariable errors
+
+        $unusable = @($errors | Where-Object { $_.Exception.Message -match 'Cannot parse the JSON schema' })
+        if ($unusable.Count -gt 0) {
+          throw 'config/persona-engine.schema.json is not valid JSON Schema. No configuration can be schema-validated until it is repaired.'
+        }
+
+        # The contract copy and the shipped copy must stay identical, or a rule author
+        # reading the contract validates against a different schema than the engine.
+        $shipped = (Get-FileHash ./config/persona-engine.schema.json -Algorithm SHA256).Hash
+        $contract = (Get-FileHash ./specs/001-persona-engine/contracts/persona-engine.schema.json -Algorithm SHA256).Hash
+
+        if ($shipped -ne $contract) {
+          throw 'config/persona-engine.schema.json and the contract copy have diverged.'
+        }
+
+  - task: PowerShell@2
+    displayName: 'Gate 5 - Example configuration passes all four layers'
+    inputs:
+      pwsh: true
+      targetType: inline
+      script: |
+        $ErrorActionPreference = 'Stop'
+        ./Edit-PersonaEngineConfig.ps1 -ConfigPath ./config/persona-engine.example.json -ValidateOnly -NonInteractive
+        if ($LASTEXITCODE -ne 0) { throw "The shipped example configuration failed validation with exit code $LASTEXITCODE." }
+
+  - task: PowerShell@2
+    displayName: 'Gate 6 - Offline Pester suite (SC-008)'
+    inputs:
+      pwsh: true
+      targetType: inline
+      script: |
+        $ErrorActionPreference = 'Stop'
+
+        $config = & ./tests/PesterConfiguration.ps1 -Suite Offline
+        $config.Run.Exit = $false
+        $config.Run.PassThru = $true
+        $config.TestResult.Enabled = $true
+        $config.TestResult.OutputPath = './testResults.offline.xml'
+        $config.Output.Verbosity = 'Normal'
+
+        $result = Invoke-Pester -Configuration $config
+
+        # Asserted, not assumed. A configuration change that silently filtered every
+        # test out would otherwise report a green pipeline over zero coverage.
+        if ($result.TotalCount -eq 0) { throw 'The offline suite ran no tests.' }
+        if ($result.FailedCount -gt 0) { throw "$($result.FailedCount) offline test(s) failed." }
+
+        Write-Host "Offline suite: $($result.PassedCount) passed, $($result.FailedCount) failed."
+
+  - task: PublishTestResults@2
+    displayName: 'Publish offline test results'
+    condition: succeededOrFailed()
+    inputs:
+      testResultsFormat: NUnit
+      testResultsFiles: './testResults.offline.xml'
+      testRunTitle: 'Persona Engine - offline suite'
+
+  - task: PowerShell@2
+    displayName: 'Gate 7 - No Graph module was loaded (SC-008)'
+    inputs:
+      pwsh: true
+      targetType: inline
+      script: |
+        # The proof that the offline suite is genuinely offline. If a test ever
+        # imports the Graph SDK, the "runs with no tenant" claim quietly stops being
+        # true and nobody notices until an air-gapped build fails.
+        $config = & ./tests/PesterConfiguration.ps1 -Suite Offline
+        $config.Run.PassThru = $true
+        $config.Output.Verbosity = 'None'
+        $null = Invoke-Pester -Configuration $config
+
+        $graph = Get-Module | Where-Object Name -like 'Microsoft.Graph*'
+        if ($graph) {
+          throw "A Graph module was loaded during the offline suite: $($graph.Name -join ', ')"
+        }
+
+        Write-Host 'No Graph module was loaded. SC-008 holds.'