fix(watchdog): suppress repeat alerts when source issue is blocked or evaluation board-closed (#5942)

## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies; the
stale-active-run watchdog monitors agent heartbeat runs for extended
output silence and fires evaluation issues to alert the responsible
manager
> - The watchdog uses a unique index on open evaluation issues to
prevent duplicate open issues per run, but this only prevents two
*simultaneous* open issues — not sequential ones created after each
closure
> - When a board reviewer closes an alert as done directly (without
recording a watchdog decision), the dismissed_false_positive guard is
bypassed and `findOpenStaleRunEvaluation` returns null on the next scan
— causing a new alert to fire every 30 minutes until the run terminates
> - The previous fix also removed `blockedByIssueIds` mutation from
`ensureSourceIssueBlockedByStaleEvaluation` to break the
block→silence→new-alert loop, but that left no idempotency guard on the
source-issue escalation comment, so every critical scan re-appended the
comment to the source-issue thread
> - Additionally, runs whose source issue is already `blocked` (agent is
correctly idle, waiting on a human action) should never generate alerts
at all — silence is expected in that state
> - This PR fixes all three gaps in `createOrUpdateStaleRunEvaluation` /
`ensureSourceIssueCommentedForStaleEvaluation`: (1) skip when source
issue is `blocked`, (2) auto-record a dismissed_false_positive decision
when a closed evaluation exists with no prior watchdog decision, (3) add
an activity-log-backed idempotency guard so the source-issue escalation
comment fires exactly once per (sourceIssue, evaluationIssue) pair
across scan cycles and process restarts
> - The benefit is that agents correctly paused waiting on board-gated
blockers no longer generate repeated false-positive noise tickets,
board-closed evaluations are permanently suppressed without requiring a
second interaction, and source-issue threads no longer get spammed with
duplicate escalation comments

## What Changed

- `server/src/services/recovery/service.ts`:
- Added `blocked` source-issue guard: `if (sourceIssue?.status ===
"blocked") return { kind: "skipped" }` — idle output is expected when
the source issue is blocked
- Added `findClosedStaleRunEvaluation()` — queries for `done` evaluation
issues for a given run, ordered by most recent update (scoped to `done`
only so system-cancelled evaluations don't permanently suppress alerts)
- Added `hasDismissedFalsePositiveDecision()` — queries for an existing
dismissed_false_positive watchdog decision record
- Added closed-evaluation auto-dismiss: when a prior evaluation was
closed `done` on the board without any watchdog decision, auto-inserts a
dismissed_false_positive record so future scans skip via the existing
guard. The check-then-insert runs inside a transaction guarded by a
per-(company, run) `pg_advisory_xact_lock` so two overlapping scans
cannot both observe `hasAnyDecision = false` and both insert duplicate
rows
- Removed `blockedByIssueIds` mutation from the escalation path and
renamed `ensureSourceIssueBlockedByStaleEvaluation` →
`ensureSourceIssueCommentedForStaleEvaluation` to reflect that the
function now only adds a comment + activity log (no state mutation) —
evaluation issues are observability-only and adding them as hard
blockers created a self-amplifying loop (blocked→silent→new
alert→blocked again)
- Added activity-log-backed idempotency guard at the top of
`ensureSourceIssueCommentedForStaleEvaluation`: query the activity log
for a `heartbeat.output_stale_escalated` row with the same (sourceIssue,
evaluationIssue) pair and return false when one is present. The single
activity-log row written on the first successful escalation is the
suppression record for all later scans, surviving process restarts
- `server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts`:
- Added: "emits the source-issue escalation comment only once across
repeated critical scans" (covers the comment-spam regression path)
  - Added: "skips ticket creation when the source issue is blocked"
- Added: "suppresses repeat alerts when evaluation is closed on the
board without a watchdog decision"
- Added: "still allows re-arm after continue decision even when
evaluation is board-closed" (exception path: if any watchdog decision
exists, human opted in to lifecycle — honour it)

## Verification

```sh
pnpm exec vitest run server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts
```

- All 18 watchdog tests pass locally
- Regression: source-issue escalation comment emits exactly once across
repeated critical scans
- Blocked source → no evaluation created (result.created === 0,
result.skipped === 1)
- Board-closed evaluation + no decisions → auto-records
dismissed_false_positive; second scan creates nothing
- Board-closed evaluation + continue decision → second scan still
creates (re-arm preserved)

## Risks

- **Low risk.** The blocked-status guard is a pure early-return that
adds no state mutation. The auto-dismiss path only inserts a record when
no decisions exist — it cannot fire for runs where a human has opted in
to the watchdog lifecycle via snooze/continue. Removing
`blockedByIssueIds` from the critical-escalation path is safe because
evaluation issues are already parented under the source issue.
- The `dismissed_false_positive` auto-insert is now race-safe under
concurrent scans via `pg_advisory_xact_lock` keyed on `(companyId,
runId)` so the check-then-insert pair is serialized without requiring a
schema change.
- `findClosedStaleRunEvaluation` is scoped to `done` only (not
`cancelled`) so system code paths that cancel evaluation issues cannot
permanently suppress future watchdog alerts for the same run.

## Model Used

- Provider: Anthropic
- Model: Claude Sonnet 4.6 (`claude-sonnet-4-6`) for original change;
Claude Opus 4.7 (`claude-opus-4-7`) for follow-up review fixes
- Context: full repo read with tool use, running as SADE agent in
Paperclip Claude Code
- Mode: agentic code analysis + targeted edit

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots (no UI changes)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

## Cross-references and status (maintainer)

- Closes #4937
- Closes #5207
- Closes #5767
- Closes #5949

Co-authored-by: Paperclip <noreply@paperclip.ing>

server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts

@@ -315,7 +315,7 @@ describeEmbeddedPostgres("active-run output watchdog", () => {
     expect(evaluation?.description).not.toContain(leakedGithubToken);
   });
 
-  it("raises critical stale-run evaluations and blocks the source issue", async () => {
+  it("raises critical stale-run evaluations with high priority but does not hard-block the source issue", async () => {
     const now = new Date("2026-04-22T20:00:00.000Z");
     const { companyId, issueId } = await seedRunningRun({
       now,
@@ -332,14 +332,126 @@ describeEmbeddedPostgres("active-run output watchdog", () => {
       .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation")));
     expect(evaluation?.priority).toBe("high");
 
-    const [blocker] = await db
+    // Evaluation issues are observability-only — they must NOT be added as hard blockers
+    // on the source issue. That caused the self-amplifying loop (KIV-1590).
+    const blockerRelations = await db
       .select()
       .from(issueRelations)
       .where(and(eq(issueRelations.companyId, companyId), eq(issueRelations.relatedIssueId, issueId)));
-    expect(blocker?.issueId).toBe(evaluation?.id);
+    expect(blockerRelations).toHaveLength(0);
 
     const [source] = await db.select().from(issues).where(eq(issues.id, issueId));
-    expect(source?.status).toBe("blocked");
+    expect(source?.status).not.toBe("blocked");
+  });
+
+  it("emits the source-issue escalation comment only once across repeated critical scans", async () => {
+    // Regression: when the same evaluation issue stays open and the watchdog re-evaluates the
+    // run as critical on every scan cycle, ensureSourceIssueCommentedForStaleEvaluation must NOT
+    // re-add the escalation comment to the source issue. Without the idempotency guard the
+    // source-issue thread is spammed once per scan cycle.
+    const now = new Date("2026-04-22T20:00:00.000Z");
+    const { companyId, issueId } = await seedRunningRun({
+      now,
+      ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000,
+    });
+    const heartbeat = heartbeatService(db);
+
+    const first = await heartbeat.scanSilentActiveRuns({ now, companyId });
+    expect(first.created).toBe(1);
+    const evaluationIssueId = first.evaluationIssueIds[0]!;
+
+    const commentsAfterFirst = await db
+      .select({ id: issueComments.id })
+      .from(issueComments)
+      .where(eq(issueComments.issueId, issueId));
+    expect(commentsAfterFirst).toHaveLength(1);
+
+    // Run the scan again at a slightly later time — the evaluation issue is still open
+    // and the run is still critical, so the existing-branch path re-invokes
+    // ensureSourceIssueCommentedForStaleEvaluation. The guard must suppress the second comment.
+    const later = new Date(now.getTime() + 60_000);
+    const second = await heartbeat.scanSilentActiveRuns({ now: later, companyId });
+    expect(second.created).toBe(0);
+
+    const commentsAfterSecond = await db
+      .select({ id: issueComments.id })
+      .from(issueComments)
+      .where(eq(issueComments.issueId, issueId));
+    expect(commentsAfterSecond).toHaveLength(1);
+
+    // Activity-log escalation entries are the persistence record — also exactly one.
+    const escalations = await db
+      .select({ id: activityLog.id })
+      .from(activityLog)
+      .where(
+        and(
+          eq(activityLog.companyId, companyId),
+          eq(activityLog.action, "heartbeat.output_stale_escalated"),
+          eq(activityLog.entityType, "issue"),
+          eq(activityLog.entityId, issueId),
+          sql`${activityLog.details} ->> 'evaluationIssueId' = ${evaluationIssueId}`,
+        ),
+      );
+    expect(escalations).toHaveLength(1);
+  });
+
+  it("skips ticket creation when the source issue is blocked (idle is expected)", async () => {
+    const now = new Date("2026-04-22T20:00:00.000Z");
+    const { companyId, issueId } = await seedRunningRun({
+      now,
+      ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000,
+    });
+    const heartbeat = heartbeatService(db);
+
+    // Mark the source issue as blocked before scanning
+    await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issueId));
+
+    const result = await heartbeat.scanSilentActiveRuns({ now, companyId });
+
+    expect(result.created).toBe(0);
+    expect(result.skipped).toBe(1);
+
+    const evaluations = await db
+      .select()
+      .from(issues)
+      .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation")));
+    expect(evaluations).toHaveLength(0);
+  });
+
+  it("does not re-file after a stale-run evaluation is dismissed as false positive", async () => {
+    const now = new Date("2026-04-22T20:00:00.000Z");
+    const { companyId, runId, managerId } = await seedRunningRun({
+      now,
+      ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000,
+    });
+    const heartbeat = heartbeatService(db);
+    const recovery = recoveryService(db, { enqueueWakeup: vi.fn() });
+
+    const first = await heartbeat.scanSilentActiveRuns({ now, companyId });
+    expect(first.created).toBe(1);
+    const evaluationIssueId = first.evaluationIssueIds[0]!;
+
+    // Reviewer records a dismissed_false_positive decision and closes the ticket
+    await recovery.recordWatchdogDecision({
+      runId,
+      actor: { type: "agent", agentId: managerId },
+      decision: "dismissed_false_positive",
+      evaluationIssueId,
+      reason: "SADE is blocked on KIV-1066 — silence is expected",
+      now,
+    });
+    await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId));
+
+    // Scan again — should not create a new ticket because of the dismissed_false_positive decision
+    const second = await heartbeat.scanSilentActiveRuns({ now, companyId });
+    expect(second.created).toBe(0);
+    expect(second.skipped).toBe(1);
+
+    const allEvaluations = await db
+      .select()
+      .from(issues)
+      .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation")));
+    expect(allEvaluations).toHaveLength(1);
   });
 
   it("folds terminal source issues with same-run durable evidence instead of creating watchdog work", async () => {
@@ -772,6 +884,81 @@ describeEmbeddedPostgres("active-run output watchdog", () => {
     ).rejects.toMatchObject({ status: 403 });
   });
 
+  it("suppresses repeat alerts when evaluation is closed on the board without a watchdog decision", async () => {
+    // Regression: KIV-1519–KIV-1618 — watchdog re-fired 18+ times for the same run because
+    // CEO closed each alert as 'done' directly on the board without recording a watchdog decision.
+    // The unique constraint only prevents duplicates while an issue is open, so once it's closed
+    // the scanner created a fresh one every cycle.
+    const now = new Date("2026-04-22T20:00:00.000Z");
+    const { companyId, runId } = await seedRunningRun({
+      now,
+      ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000,
+    });
+    const heartbeat = heartbeatService(db);
+
+    // First scan: creates the evaluation issue.
+    const first = await heartbeat.scanSilentActiveRuns({ now, companyId });
+    expect(first.created).toBe(1);
+    const evaluationIssueId = first.evaluationIssueIds[0];
+    expect(evaluationIssueId).toBeTruthy();
+
+    // Reviewer closes the issue on the board — no watchdog decision recorded.
+    await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId));
+
+    // Second scan: should NOT create a new issue. Instead auto-records dismissed_false_positive.
+    const second = await heartbeat.scanSilentActiveRuns({ now, companyId });
+    expect(second.created).toBe(0);
+    expect(second.skipped).toBe(1);
+
+    // All subsequent scans also skip.
+    const third = await heartbeat.scanSilentActiveRuns({ now, companyId });
+    expect(third.created).toBe(0);
+    expect(third.skipped).toBe(1);
+
+    // The auto-recorded dismissed_false_positive decision must be persisted.
+    const decisions = await db
+      .select()
+      .from(heartbeatRunWatchdogDecisions)
+      .where(and(eq(heartbeatRunWatchdogDecisions.runId, runId), eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive")));
+    expect(decisions).toHaveLength(1);
+    expect(decisions[0].evaluationIssueId).toBe(evaluationIssueId);
+  });
+
+  it("still allows re-arm after continue decision even when issue was closed on board", async () => {
+    // When a human explicitly recorded a 'continue' decision the watchdog lifecycle is active;
+    // closing the issue on the board should not permanently suppress future alerts.
+    const now = new Date("2026-04-22T20:00:00.000Z");
+    const { companyId, managerId, runId } = await seedRunningRun({
+      now,
+      ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000,
+    });
+    const heartbeat = heartbeatService(db);
+    const recovery = recoveryService(db, { enqueueWakeup: vi.fn() });
+
+    const scan = await heartbeat.scanSilentActiveRuns({ now, companyId });
+    const evaluationIssueId = scan.evaluationIssueIds[0];
+    expect(evaluationIssueId).toBeTruthy();
+
+    // Human records a 'continue' decision.
+    await recovery.recordWatchdogDecision({
+      runId,
+      actor: { type: "agent", agentId: managerId },
+      decision: "continue",
+      evaluationIssueId,
+      reason: "Expected quiet period.",
+      now,
+    });
+
+    // Board also closes the evaluation issue.
+    await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId));
+
+    // After the re-arm window, watchdog should still fire a new alert.
+    const rearmAt = new Date(now.getTime() + ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS + 60_000);
+    const rearm = await heartbeat.scanSilentActiveRuns({ now: rearmAt, companyId });
+    expect(rearm.created).toBe(1);
+    expect(rearm.evaluationIssueIds[0]).not.toBe(evaluationIssueId);
+  });
+
   it("validates createdByRunId before storing watchdog decisions", async () => {
     const now = new Date("2026-04-22T20:00:00.000Z");
     const { companyId, managerId, runId } = await seedRunningRun({