Compatibility
| Surface |
Status |
| Harness |
DeepSeek Harness dsh-v0.1.5-rc.1 (GitHub tag, verified 2026-09-10). Dual-line npm support: dev pins and runtime deps 0.1.5-rc.1, peers >=0.1.2-rc.1 <0.2.0 || >=0.1.5-alpha.1 <0.2.0 — the plugin code feature-detects both published host lines and each line runs the full gate chain; the runtime dependency pins follow the alpha line so a profile install never shadows the host's own 0.1.5-rc.1 tree. |
| Node |
^22.19.0 || >=24.0.0 |
| Platforms |
All (host answerer; optional Web review panel via the session-projection capability) |
| Model |
Any (the reviewer inherits the session agent's route; reviewerModel overrides) |
What you get
dsh-auto-review puts a second model on the approval/request answerer chain:
- Official seam — an answerer that claims only the requests it owns (
ai policy) and delegates everything else via next(); the human approval flow is never short-circuited.
- Read-only reviewer subagent — a one-shot fork with a
read/glob/grep tool allow-list returns a structured verdict { decision, reason, riskLevel }. Reviewer asks are recognized by identity and delegated; maxDepth + the allow-list keep the reviewer non-delegating.
- Fail closed — reviewer crash, timeout, or schema mismatch resolves through
fallbackPolicy (default rejected); a deny verdict feeds its reason back to the calling model.
- Config-driven routing — per-tool policies (
ai/human/never) plus regex risk rules, all changeable from cordis.yml.
- Deny reasons reach the model — the reviewer's reason is injected into the denied tool result (callId-linked); fallback and
never-policy rejections inject auditable markers too ([auto-review] / [auto-review-fallback] / [auto-review-never]).
- Full audit trail — log-only
autoReview/verdict + autoReview/rejection session events (envelope ignorable: true) plus an optional invariant companion enforcing marker ⟺ event.
- Safety knobs — a rejection circuit breaker (3 consecutive denials, or 6 of the last 10 verdicts, per turn), a risk-level policy, a one-shot
/auto-review approve override, and a never-policy hard disable that explains itself to the model.
- Optional reviewer context — a bounded compact transcript (
contextBudget) plus a Codex-style Markdown ruling policy (reviewerPolicyText).
Every decision reconstructs from the session log: approval/asked → autoReview/verdict (or autoReview/rejection) → approval/decided.
Why a second model instead of rules?
Pattern-based auto-approvers decide before dispatch, with no evidence. dsh-auto-review gives the decision to a reviewer subagent that reads the actual workspace (through its read-only tool face), the already-streamed tool-call arguments (sensitive values redacted), the request reason, and your risk rules — then returns a structured verdict. A deny verdict feeds its reason back to the calling model, so the agent learns why instead of retrying blindly.
Quick start
# 1. install the bundle into your profile
dsh plugin --profile web add "github:PerryLink/dsh-auto-review#main"
# or from npm (published releases)
dsh plugin --profile web add dsh-auto-review
# 2. restart and verify the row
dsh --profile web --dump-config | grep -A4 'id: auto-review'
Out of the box the shipped patch AI-reviews bash and write; every other tool (including edit — in-place modification) delegates to the human chain. Add edit: ai explicitly if you accept in-place edits without a human in the loop.
Install & uninstall
- git channel (latest
main): dsh plugin --profile web add "github:PerryLink/dsh-auto-review#main" — the isolated prepare build needs the single allowBuilds: { esbuild: true } key the dsh CLI prints for dsh-auto-review.
- npm channel (published releases):
dsh plugin --profile web add dsh-auto-review.
- 1024 store channel:
npm i -g dsh1024 once, then dsh1024 plugin --profile web add dsh-auto-review (counts toward the deepseek1024.com install ranking).
- tarball channel:
pnpm pack in this repo, then dsh plugin --profile web add ./dsh-auto-review-<version>.tgz.
- uninstall:
dsh plugin --profile web remove dsh-auto-review (or remove the row from the profile patch).
- native build scripts: when
dsh plugin add stops at ERR_PNPM_IGNORED_BUILDS for koffi / node-pty (pulled in by the eval harness), run pnpm approve-builds to approve those build scripts.
Configuration
All tunables are Schemastery Config fields (changeable from cordis.yml). An id-targeted override replaces the whole row — restate every key you need.
| Key |
Default |
Meaning |
enableByDefault |
true |
Sessions start with auto-review enabled; /auto-review on|off writes a durable override that beats this |
toolsPolicy.default |
human |
Policy for unlisted tools (delegate to the human answerer) |
toolsPolicy.overrides |
{} |
Per-tool policy: ai / human / never |
riskRules |
[] |
{pattern, policy, field?} matched before the tool table; field selects reason (default), toolName, or arguments |
reviewerProvider |
fork |
Subagent provider for the reviewer (in-process fork backend) |
reviewerModel |
(inherit) |
Reviewer model id; unset inherits the session agent's route |
reviewerTimeoutMs |
60000 |
Verdict deadline; on expiry the fallback policy applies |
reviewerTools |
[read, glob, grep] |
The reviewer child's tool allow-list (must be non-empty) |
fallbackPolicy |
rejected |
Reviewer failure: rejected (fail closed) / delegate / allow-once |
maxReviewsPerTurn |
10 |
Real AI-verdict budget per open turn; beyond it, requests delegate |
maxFailuresPerTurn |
10 |
Reviewer-failure budget per open turn |
reasonMaxChars |
2000 |
Cap for reviewer reasons and the redacted argument preview |
reviewerGuidance |
(none) |
Optional advisory guidance appended to the reviewer prompt |
reviewerPolicyText |
(none) |
Markdown ruling policy injected into the reviewer prompt (Codex-style) |
denyGuidance |
(anti-circumvention text) |
Guidance appended to every injected deny reason |
contextBudget |
{turns: 2, maxChars: 4000} |
Compact transcript budget for the reviewer prompt (the open turn plus the one before it); turns: 0 disables the section — and a blind reviewer denies user-authorized actions, so the runtime warns when 0 meets an ai policy. The character budget is spent on the most recent lines |
riskPolicy |
{maxAutoAllow: high, onHighRisk: delegate} |
allow verdicts above maxAutoAllow delegate or deny |
circuitBreaker |
{consecutiveDenies: 3, windowDenies: 6, windowSize: 10, action: delegate} |
Rejection circuit breaker |
overrideTtlMs |
300000 |
How long a /auto-review approve override stays usable |
verdictCacheTtlMs |
60000 |
Reuse a recent verdict for an identical tool + arguments fingerprint; 0 disables the cache. Only applies with contextBudget.turns: 0 — a transcript-dependent verdict is not replayable from tool + arguments alone |
verdictCacheMaxEntries |
256 |
Maximum cached fingerprints before oldest-eviction |
language |
en |
UI language of the /auto-review command output (en | zh) |
allowUnmarkedAudit |
false |
Force session-log audit on hosts that drop the ignorable marker or fail-closed on unknown event types (host 0.1.2-rc.1+) (dangerous: unmarked events make sessions unresumable elsewhere); default is detect-and-degrade (adapted 2026-09-02, re-verified against 0.1.5-rc.1 on 2026-09-10): the session envelope keeps its ignorable field for stored-log read compatibility only - Session.append still cannot stamp it, so audit-gate behavior is unchanged. |
Example (annotated full form: fixtures/config/config-full.yaml):
- insert:
- id: auto-review
name: dsh-auto-review
config:
toolsPolicy:
overrides: { bash: ai, write: ai }
riskRules:
- pattern: '(?i)(rm\s+(-[a-z]+\s+)*/|git\s+push\s+--force)'
policy: never
- pattern: 'write'
policy: never
field: toolName
reviewerTimeoutMs: 30000
fallbackPolicy: delegate
riskPolicy: { maxAutoAllow: medium, onHighRisk: delegate }
circuitBreaker: { consecutiveDenies: 3, windowDenies: 6, windowSize: 10, action: delegate }
Where the config actually comes from
~/.dsh/settings.yaml is NOT a config source for this plugin. An auto-review: block there has no effect and produces no warning: like every DSH function plugin, dsh-auto-review receives its Config from the row the loader mounts it with — the profile's cordis patch layer. (Some other DSH plugins additionally read the settings service, so the inconsistency is easy to trip over, and the symptom is indistinguishable from the reviewer simply denying.)
Put the configuration in your profile's cordis.patch.yml. An id-targeted override replaces the whole config row, so restate every key you need — dropping toolsPolicy silently returns bash/write to the schema default human and the reviewer stops running at all:
- id: auto-review
config:
toolsPolicy:
overrides: { bash: ai, write: ai }
contextBudget: { turns: 4, maxChars: 8000 }
| Surface |
Kind |
Notes |
auto-review |
answerer |
approval/request waterfall answerer — claims ai-policy requests, delegates the rest via next() |
/auto-review |
command |
on|off|status|approve [n] — durable per-session override, budgets, and cumulative statistics |
| deny-reason injection |
listener |
tools/post-execute — verdict / fallback / never reasons fed back to the denied tool result |
autoReview |
session projection |
Folded from the log-only autoReview/* events |
| Web review panel |
client |
Session-header action: switch, budgets, statistics, recent verdicts, one-shot approve |
dsh-eval |
CLI |
YAML-driven agent evaluation engine (bin/dsh-eval.mjs) |
| invariant companion |
invariant |
dsh-auto-review/invariant (optional; needs the invariants service) |
Session command
/auto-review on|off|status|approve [n]
on/off append the durable autoReview/state override (the fold survives restart/resume — replay IS the state) and inject a switch notice the model sees (logged as a user/message event). status reports the effective state, both per-turn budgets (AI verdicts and reviewer failures), a tripped circuit breaker when one is active, and the session's cumulative statistics (allows/denies/fallbacks/never rejects, mean duration, recent verdicts). approve [n] records a single-use autoReview/override for the n-th most recent denial (1 = most recent): the next same-tool review within overrideTtlMs carries the authorization as reviewer context — the reviewer still decides, and the override is consumed by that review regardless of its outcome.
Web review panel
In the Web GUI (web profile), the package contributes a session-header action (AI Review) that opens a panel with the session's auto-review state: the switch with on/off buttons (they execute /auto-review on|off), both per-turn budgets, cumulative statistics (including hard-disable rejections and cache hits), the circuit trip, the recent verdicts, and one-shot approve buttons for recent denials (they execute /auto-review approve [n]).
How it is wired:
- The host registers an
autoReview session projection (folded from the log-only autoReview/* events) and serves it through the session-projection channel.
- The browser half is a client module (auto-discovered from the
dsh.client declaration) registered on the conversation.session.header.actions seat.
- No extra patch rows are needed: the panel loads whenever the plugin is installed in a profile whose web build provides the session-projection capability (the web profile does). Without that capability the panel reports itself unavailable; the answerer is unaffected.
The panel reads only whole projection values — it never receives the raw session event stream.
How it works
approval/request waterfall (answerer chain)
│
┌───────────────────────┴──────────────────────┐
│ dsh-auto-review answerer │
│ · session enabled? · policy = ai? │ no ── next() ──▶ human answerer (UI)
│ · risk rules → toolsPolicy → default │
└───────────────────────┬──────────────────────┘
│ yes
▼
┌───────────────────────────────────┐
│ reviewer subagent (fork, one-shot)│
│ · toolFilter: read/glob/grep │
│ · outputSchema: {decision, │
│ reason, riskLevel} │
│ · timeout + req.signal abort │
└───────────────┬───────────────────┘
│ verdict / failure (fail-closed fallback)
▼
allow → allowed-once deny → rejected + reason injected into the
denied tool result (callId-linked)
│ never → rejected + [auto-review-never] feedback
│ (hard disable, no reviewer runs)
▼
audit: approval/asked → autoReview/verdict | autoReview/rejection
→ approval/decided (session events, log-only, invariant-checked)
Composition order. The answerer runs at its registration position in the waterfall: if a human UI answerer is composed BEFORE the auto-review row, humans answer first and the reviewer only sees what is delegated downstream. Verify with dsh --profile <name> --dump-config and place the auto-review row before your human answerer rows when you want ai-policy tools routed to the reviewer first.
dsh-eval — agent evaluation engine
Beyond the approval reviewer, dsh-auto-review ships dsh-eval: a YAML-driven agent evaluation platform that runs real headless DSH sessions (one isolated agent + scratch workspace per case, the official Minimal persona as the baseline system prompt), collects the tool-call trace from the session event log, and evaluates structured assertions plus an optional second-model review — the same reviewer seam as the approval answerer.
# eval/cases/demo.yaml (abridged)
suite:
name: my-suite
cases:
- id: math-output
input: Solve 17 × 24 and reply with only the final number, nothing else.
expect:
output: { contains: "408" }
- id: glob-trace
seedFrom: '.'
input: Use the glob tool with pattern "src/**" to list the source files…
expect:
toolCalls: [{ tool: glob, arguments: { contains: { pattern: "src" } } }]
results: [{ tool: glob, contains: "index.ts" }]
Run it (a DeepSeek API key must be in the environment):
dsh-eval eval/cases --model deepseek-v4-flash --timeout-ms 240000 --out .eval-reports
Assertion families
The expect block supports six assertion families; each assertion is evaluated independently and reports its own pass/fail with expected/actual values, so a failing case explains itself without a rerun.
| Family |
DSL keys |
What it gates |
| Tool trace |
toolCalls, toolCallsExact, noToolCalls, results |
ordered tool-call sequence (subsequence with skips), exact name sequence, per-tool result (isError/contains/regex) |
| Output & budget |
output, turnEnds, maxTokens |
final-output substring/regex, turn outcome, token budget |
| Prompt regression |
prompt |
the rendered system prompt must match a committed baseline (or a baselineFrom file); any drift is reported as a side-by-side diff, with allowedChanges regexes to whitelist intended edits |
| Stress metrics |
stress |
P99 step latency (maxP99Ms), worst time-to-first-token (maxTtftMs), aggregate token generation speed (minTokensPerSecond) |
| Fairness |
bias |
bias radar over the final output: per-category regex counts (categories), hard forbid patterns, maxHits/maxCategoryHits caps |
| Second-model review |
review |
a supplementary pass/fail verdict from the reviewer subagent (a separate layer, same seam as the approval reviewer) |
- id: regression-gate
input: Answer in one sentence.
expect:
prompt:
baseline: "You are a helpful software engineer assistant."
allowedChanges: ["copyright-year"]
stress:
maxP99Ms: 8000
maxTtftMs: 3000
minTokensPerSecond: 20
bias:
categories: { gender: ["[Hh]e is (un)?stable"] }
forbid: ["[Ss]crew that"]
maxCategoryHits: 0
CI gate: the process exits 0 only when every case of every suite passed — failing evaluations fail the build. Each case leaves a replayable session JSONL and a trace JSON beside report.md/report.json; assertion results (including the prompt side-by-side diff), token usage, stress/bias metrics, and the review verdict are all written into the report files.
- name: dsh-eval
run: npx dsh-eval eval/cases --model deepseek-v4-flash --timeout-ms 240000 --out .eval-reports
env:
DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
dsh-eval differs from openai/codex-research: codex-research scores agent trajectories for research comparison; dsh-eval is a declarative pass/fail regression harness — YAML case
A second-model AI auto-reviewer plugin for DeepSeek Harness: a read-only reviewer subagent decides allow/deny on tool-call approval requests, fail-closed by default, with a full audit trail.