Safety built in, not bolted on.
Autonomous AI development only works when something keeps it honest. Nightgauge ships with deterministic guardrails at every layer — from per-stage verification to org-wide circuit breakers — so the pipeline can run unattended without becoming a liability.
The autonomous safety stack
Each layer sits at a different point in the pipeline — together they make the difference between "an AI tool" and "an AI dev system you can leave running overnight."
Cascade Circuit Breaker
A sliding-window failure tracker that halts autonomous mode the moment failures start to cluster — so a systemic fault never fans out across your repo.
Deterministic Stage Gates
After every LLM stage, a deterministic post-condition check verifies the work actually happened. No more 'skill said success but didn't do the work.'
Prompt Injection Firewall
Two-layer input/output validation blocks dangerous bash patterns, credential exfiltration, path traversal, and privilege escalation before they ever execute.
Failure Taxonomy
Weighted classification distinguishes transient infra failures (rate limits, network blips) from organic ones — so health scores aren't polluted by noise.
Audit Archival
Every stage, every fix, every recovery — captured as a structured event stream with configurable retention. Built for the compliance review you haven't been asked for yet.
Baseline-Failure Detection
Pre-existing CI failures are tracked separately from new ones. The auto- fix loop only addresses regressions your change introduced — never legacy flakes.
Cascade Circuit Breaker
Autonomous pipelines fail fast and quietly when something is wrong. Without a backstop, a single bad input can fan out into a cascading wave of broken PRs. The circuit breaker is that backstop.
Sliding window watches every autonomous run, across repos. Default: 3 failures in 30 minutes trips the breaker.
Autonomous mode is paused immediately. In-flight stages complete; new pipelines do not start. A diagnostic event is emitted with the failure cluster.
No automatic resume. The breaker waits for a human to inspect the cluster, fix the upstream issue, and explicitly re-arm. Safer than eventual self-healing.
Stall kills, quota exhaustion and budget-ceiling stops don’t count toward the threshold, so legitimate retries never burn it down.
Trip events post to Discord and show in VS Code, listing every involved issue and how it failed — no log-grepping required.
Two environment variables override the threshold and the window. A malformed value falls back to the default instead of breaking the autonomous loop.
Cost Caps, Budgets & Timeouts
A stuck agent doesn’t announce itself — it keeps spending, or it goes quiet. Deterministic ceilings on cost, progress and time stop a runaway stage whatever the model believes about its own work.
Cost caps
A hard dollar ceiling with no grace and no prompt: cross it and the agent is terminated. Separately, a budget enforcer compares estimated against actual spend for the whole run, with a grace buffer.
- Checked on every streaming usage update, so overshoot is bounded to a single tool call
- A cost-cap kill is never retried
Forward-progress budget
A stage is killed only when no new progress signal arrives inside a sliding window — so active-but-expensive work runs to completion, while a cheap infinite loop is stopped quickly.
- Progress means phase markers, file changes, commits, CI progress and distinct tool calls
- Repeating the same call doesn’t count, which catches the read-the-same-file loop
- On by default: a 2-minute window, armed once a stage spends $0.50
Stall detection
Idle time is tracked apart from elapsed time. A stage that goes quiet gets escalating stall warnings, then a nudge, then a kill — an agent waiting on something that will never arrive doesn’t wait forever.
- Warning thresholds set per stage, from 1 minute to 10 minutes
- Killed at 8× the threshold by default, after a 60-second grace
Timeouts
Stages run under wall-clock, turn and token ceilings — by default 4 hours, 400 turns and 25 million tokens. The defaults sit above what a normal stage uses, so they only stop runaways.
- Unlimited is explicit: only
-1lifts a ceiling, and every run with one lifted logs a warning - Local models priced at $0, which no dollar cap can stop, get a 200-turn default and can’t lift the ceilings
None of these asks a model whether to fire. One trade-off to know: in Maximum performance mode, the forward-progress kill and the pipeline’s token budget ceiling warn instead of stopping the run.
Deterministic Stage Gates & Failure Taxonomy
Trust LLM output, but verify deterministically. After every stage, Nightgauge runs a hard post-condition check, and gate phases inside the stages verify builds, baselines and scope along the way. Every failure is weighed so transient noise doesn’t look like a regression.
Post-condition gates
Each stage declares the artifacts it promised to produce — a plan file, a passing build, a created PR. A deterministic checker runs after the skill returns. Skill says success ≠ stage passes.
- Plan stage →
PLAN.mdexists and has acceptance criteria - Dev stage →
git diffshows actual changes - Validate stage → build & tests actually ran
- PR-create stage → PR exists, links the issue
Weighted failure taxonomy
Not every failure is the same. Infra failures (rate limits, network blips, GitHub API timeouts) are weighted differently from organic failures (bad code, broken tests, missed acceptance criteria).
- Health score isn’t polluted by transient infra noise
- Retry policy adapts to the failure class
- Circuit breaker only counts organic failures toward its threshold
Baseline failure detection
Before any auto-fix runs, the pipeline records the baseline of CI failures already present on the branch. The auto-fix loop only attempts to resolve failures the current change introduced.
- No wasted tokens fixing legacy flakes
- RALPH loop converges on real regressions
- False-alarm rate dropped 80%+ since #3662
Stage-exit diagnostic
Every stop — clean or otherwise — writes a structured exit record: last bash command, stderr tail, rate-limit signals, stall reason. The retro skill reads these to root-cause incidents in seconds, not hours.
Prompt Injection Firewall
LLMs read everything in your codebase — including malicious comments and crafted issue bodies. A two-layer firewall validates both what the model receives and what it tries to run.
Input sanitization
Issue bodies, PR descriptions, file contents, and tool outputs are scanned for injection patterns before they reach the model. Suspicious payloads are flagged or stripped.
- Hidden instruction overrides ("ignore previous", "act as")
- Obfuscated bash via base64 / unicode tricks
- Credential exfiltration patterns
- Tool-call hijacking attempts
Output validation
Every bash command, file write, and tool call from the model passes through a deterministic checker before execution. Dangerous patterns are blocked outright, not just warned.
- Bash escapes and process-substitution tricks
- Path traversal beyond the workspace
- Privilege escalation (
sudo,chmod 777) - Network egress to disallowed hosts
Plus the rest of the standard security playbook
PR pre-flight blocks any commit containing recognisable API keys, tokens, or PEM blocks.
Pipelines cannot read or write outside the declared workspace root.
No reaching into ~/.ssh.
Context is JSON-handoff between stages — no conversation history leaks between issues.
The core — extension, orchestrator, and skills — is Apache-2.0 on GitHub. Every guardrail on this page is open source, not a black box.
Audit Archival
Every pipeline run produces an immutable trail. Every stage start and exit, every retry, every fix, every cost line item — captured as a structured event stream you can query, export, and retain for as long as compliance requires.
Structured events at every state transition: stage-enter, stage-exit, retry, fix, gate-failure, escalation, model-switch, cost-event.
Configurable retention per account (90d minimum, 2y default), then hash-chain-preserving archival to cold storage.
Tail-the-pipeline view, time-range queries, JSON / CSV export. The Audit dashboard tab is your compliance review.
What the audit trail enables
- Post-incident retro reads exit records to root-cause in seconds
- Cost attribution per issue, per epic, per repo, per developer
- SOC 2 / ISO control evidence on AI-assisted code changes
- Diff-review across pipeline runs to spot drift
- Per-PR provenance: every PR links back to the pipeline run, stages, models, and prompts that produced it
Together — circuit breaker, budgets and timeouts, stage gates, failure taxonomy, baseline detection, injection firewall, and audit archival — these layers are what let teams point Nightgauge at a repo and walk away. The pipeline isn’t just smart. It’s governed.
Ready to see it in action?
The Nightgauge core is open source under Apache-2.0. Install the VS Code extension from the VS Code Marketplace or Open VSX, or explore the code on GitHub.