fix(watchdog): suppress repeat alerts when source issue is blocked or evaluation board-closed (#5942)

## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies; the
stale-active-run watchdog monitors agent heartbeat runs for extended
output silence and fires evaluation issues to alert the responsible
manager
> - The watchdog uses a unique index on open evaluation issues to
prevent duplicate open issues per run, but this only prevents two
*simultaneous* open issues — not sequential ones created after each
closure
> - When a board reviewer closes an alert as done directly (without
recording a watchdog decision), the dismissed_false_positive guard is
bypassed and `findOpenStaleRunEvaluation` returns null on the next scan
— causing a new alert to fire every 30 minutes until the run terminates
> - The previous fix also removed `blockedByIssueIds` mutation from
`ensureSourceIssueBlockedByStaleEvaluation` to break the
block→silence→new-alert loop, but that left no idempotency guard on the
source-issue escalation comment, so every critical scan re-appended the
comment to the source-issue thread
> - Additionally, runs whose source issue is already `blocked` (agent is
correctly idle, waiting on a human action) should never generate alerts
at all — silence is expected in that state
> - This PR fixes all three gaps in `createOrUpdateStaleRunEvaluation` /
`ensureSourceIssueCommentedForStaleEvaluation`: (1) skip when source
issue is `blocked`, (2) auto-record a dismissed_false_positive decision
when a closed evaluation exists with no prior watchdog decision, (3) add
an activity-log-backed idempotency guard so the source-issue escalation
comment fires exactly once per (sourceIssue, evaluationIssue) pair
across scan cycles and process restarts
> - The benefit is that agents correctly paused waiting on board-gated
blockers no longer generate repeated false-positive noise tickets,
board-closed evaluations are permanently suppressed without requiring a
second interaction, and source-issue threads no longer get spammed with
duplicate escalation comments

## What Changed

- `server/src/services/recovery/service.ts`:
- Added `blocked` source-issue guard: `if (sourceIssue?.status ===
"blocked") return { kind: "skipped" }` — idle output is expected when
the source issue is blocked
- Added `findClosedStaleRunEvaluation()` — queries for `done` evaluation
issues for a given run, ordered by most recent update (scoped to `done`
only so system-cancelled evaluations don't permanently suppress alerts)
- Added `hasDismissedFalsePositiveDecision()` — queries for an existing
dismissed_false_positive watchdog decision record
- Added closed-evaluation auto-dismiss: when a prior evaluation was
closed `done` on the board without any watchdog decision, auto-inserts a
dismissed_false_positive record so future scans skip via the existing
guard. The check-then-insert runs inside a transaction guarded by a
per-(company, run) `pg_advisory_xact_lock` so two overlapping scans
cannot both observe `hasAnyDecision = false` and both insert duplicate
rows
- Removed `blockedByIssueIds` mutation from the escalation path and
renamed `ensureSourceIssueBlockedByStaleEvaluation` →
`ensureSourceIssueCommentedForStaleEvaluation` to reflect that the
function now only adds a comment + activity log (no state mutation) —
evaluation issues are observability-only and adding them as hard
blockers created a self-amplifying loop (blocked→silent→new
alert→blocked again)
- Added activity-log-backed idempotency guard at the top of
`ensureSourceIssueCommentedForStaleEvaluation`: query the activity log
for a `heartbeat.output_stale_escalated` row with the same (sourceIssue,
evaluationIssue) pair and return false when one is present. The single
activity-log row written on the first successful escalation is the
suppression record for all later scans, surviving process restarts
- `server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts`:
- Added: "emits the source-issue escalation comment only once across
repeated critical scans" (covers the comment-spam regression path)
  - Added: "skips ticket creation when the source issue is blocked"
- Added: "suppresses repeat alerts when evaluation is closed on the
board without a watchdog decision"
- Added: "still allows re-arm after continue decision even when
evaluation is board-closed" (exception path: if any watchdog decision
exists, human opted in to lifecycle — honour it)

## Verification

```sh
pnpm exec vitest run server/src/__tests__/heartbeat-active-run-output-watchdog.test.ts
```

- All 18 watchdog tests pass locally
- Regression: source-issue escalation comment emits exactly once across
repeated critical scans
- Blocked source → no evaluation created (result.created === 0,
result.skipped === 1)
- Board-closed evaluation + no decisions → auto-records
dismissed_false_positive; second scan creates nothing
- Board-closed evaluation + continue decision → second scan still
creates (re-arm preserved)

## Risks

- **Low risk.** The blocked-status guard is a pure early-return that
adds no state mutation. The auto-dismiss path only inserts a record when
no decisions exist — it cannot fire for runs where a human has opted in
to the watchdog lifecycle via snooze/continue. Removing
`blockedByIssueIds` from the critical-escalation path is safe because
evaluation issues are already parented under the source issue.
- The `dismissed_false_positive` auto-insert is now race-safe under
concurrent scans via `pg_advisory_xact_lock` keyed on `(companyId,
runId)` so the check-then-insert pair is serialized without requiring a
schema change.
- `findClosedStaleRunEvaluation` is scoped to `done` only (not
`cancelled`) so system code paths that cancel evaluation issues cannot
permanently suppress future watchdog alerts for the same run.

## Model Used

- Provider: Anthropic
- Model: Claude Sonnet 4.6 (`claude-sonnet-4-6`) for original change;
Claude Opus 4.7 (`claude-opus-4-7`) for follow-up review fixes
- Context: full repo read with tool use, running as SADE agent in
Paperclip Claude Code
- Mode: agentic code analysis + targeted edit

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots (no UI changes)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

## Cross-references and status (maintainer)

- Closes #4937
- Closes #5207
- Closes #5767
- Closes #5949

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
David Bezar
2026-06-11 22:00:13 -07:00
committed by GitHub
parent deef1f479d
commit 01e59c074a
2 changed files with 323 additions and 17 deletions
@@ -315,7 +315,7 @@ describeEmbeddedPostgres("active-run output watchdog", () => {
expect(evaluation?.description).not.toContain(leakedGithubToken); expect(evaluation?.description).not.toContain(leakedGithubToken);
}); });
it("raises critical stale-run evaluations and blocks the source issue", async () => { it("raises critical stale-run evaluations with high priority but does not hard-block the source issue", async () => {
const now = new Date("2026-04-22T20:00:00.000Z"); const now = new Date("2026-04-22T20:00:00.000Z");
const { companyId, issueId } = await seedRunningRun({ const { companyId, issueId } = await seedRunningRun({
now, now,
@@ -332,14 +332,126 @@ describeEmbeddedPostgres("active-run output watchdog", () => {
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation"))); .where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation")));
expect(evaluation?.priority).toBe("high"); expect(evaluation?.priority).toBe("high");
const [blocker] = await db // Evaluation issues are observability-only — they must NOT be added as hard blockers
// on the source issue. That caused the self-amplifying loop (KIV-1590).
const blockerRelations = await db
.select() .select()
.from(issueRelations) .from(issueRelations)
.where(and(eq(issueRelations.companyId, companyId), eq(issueRelations.relatedIssueId, issueId))); .where(and(eq(issueRelations.companyId, companyId), eq(issueRelations.relatedIssueId, issueId)));
expect(blocker?.issueId).toBe(evaluation?.id); expect(blockerRelations).toHaveLength(0);
const [source] = await db.select().from(issues).where(eq(issues.id, issueId)); const [source] = await db.select().from(issues).where(eq(issues.id, issueId));
expect(source?.status).toBe("blocked"); expect(source?.status).not.toBe("blocked");
});
it("emits the source-issue escalation comment only once across repeated critical scans", async () => {
// Regression: when the same evaluation issue stays open and the watchdog re-evaluates the
// run as critical on every scan cycle, ensureSourceIssueCommentedForStaleEvaluation must NOT
// re-add the escalation comment to the source issue. Without the idempotency guard the
// source-issue thread is spammed once per scan cycle.
const now = new Date("2026-04-22T20:00:00.000Z");
const { companyId, issueId } = await seedRunningRun({
now,
ageMs: ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS + 60_000,
});
const heartbeat = heartbeatService(db);
const first = await heartbeat.scanSilentActiveRuns({ now, companyId });
expect(first.created).toBe(1);
const evaluationIssueId = first.evaluationIssueIds[0]!;
const commentsAfterFirst = await db
.select({ id: issueComments.id })
.from(issueComments)
.where(eq(issueComments.issueId, issueId));
expect(commentsAfterFirst).toHaveLength(1);
// Run the scan again at a slightly later time — the evaluation issue is still open
// and the run is still critical, so the existing-branch path re-invokes
// ensureSourceIssueCommentedForStaleEvaluation. The guard must suppress the second comment.
const later = new Date(now.getTime() + 60_000);
const second = await heartbeat.scanSilentActiveRuns({ now: later, companyId });
expect(second.created).toBe(0);
const commentsAfterSecond = await db
.select({ id: issueComments.id })
.from(issueComments)
.where(eq(issueComments.issueId, issueId));
expect(commentsAfterSecond).toHaveLength(1);
// Activity-log escalation entries are the persistence record — also exactly one.
const escalations = await db
.select({ id: activityLog.id })
.from(activityLog)
.where(
and(
eq(activityLog.companyId, companyId),
eq(activityLog.action, "heartbeat.output_stale_escalated"),
eq(activityLog.entityType, "issue"),
eq(activityLog.entityId, issueId),
sql`${activityLog.details} ->> 'evaluationIssueId' = ${evaluationIssueId}`,
),
);
expect(escalations).toHaveLength(1);
});
it("skips ticket creation when the source issue is blocked (idle is expected)", async () => {
const now = new Date("2026-04-22T20:00:00.000Z");
const { companyId, issueId } = await seedRunningRun({
now,
ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000,
});
const heartbeat = heartbeatService(db);
// Mark the source issue as blocked before scanning
await db.update(issues).set({ status: "blocked" }).where(eq(issues.id, issueId));
const result = await heartbeat.scanSilentActiveRuns({ now, companyId });
expect(result.created).toBe(0);
expect(result.skipped).toBe(1);
const evaluations = await db
.select()
.from(issues)
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation")));
expect(evaluations).toHaveLength(0);
});
it("does not re-file after a stale-run evaluation is dismissed as false positive", async () => {
const now = new Date("2026-04-22T20:00:00.000Z");
const { companyId, runId, managerId } = await seedRunningRun({
now,
ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000,
});
const heartbeat = heartbeatService(db);
const recovery = recoveryService(db, { enqueueWakeup: vi.fn() });
const first = await heartbeat.scanSilentActiveRuns({ now, companyId });
expect(first.created).toBe(1);
const evaluationIssueId = first.evaluationIssueIds[0]!;
// Reviewer records a dismissed_false_positive decision and closes the ticket
await recovery.recordWatchdogDecision({
runId,
actor: { type: "agent", agentId: managerId },
decision: "dismissed_false_positive",
evaluationIssueId,
reason: "SADE is blocked on KIV-1066 — silence is expected",
now,
});
await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId));
// Scan again — should not create a new ticket because of the dismissed_false_positive decision
const second = await heartbeat.scanSilentActiveRuns({ now, companyId });
expect(second.created).toBe(0);
expect(second.skipped).toBe(1);
const allEvaluations = await db
.select()
.from(issues)
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "stale_active_run_evaluation")));
expect(allEvaluations).toHaveLength(1);
}); });
it("folds terminal source issues with same-run durable evidence instead of creating watchdog work", async () => { it("folds terminal source issues with same-run durable evidence instead of creating watchdog work", async () => {
@@ -772,6 +884,81 @@ describeEmbeddedPostgres("active-run output watchdog", () => {
).rejects.toMatchObject({ status: 403 }); ).rejects.toMatchObject({ status: 403 });
}); });
it("suppresses repeat alerts when evaluation is closed on the board without a watchdog decision", async () => {
// Regression: KIV-1519KIV-1618 — watchdog re-fired 18+ times for the same run because
// CEO closed each alert as 'done' directly on the board without recording a watchdog decision.
// The unique constraint only prevents duplicates while an issue is open, so once it's closed
// the scanner created a fresh one every cycle.
const now = new Date("2026-04-22T20:00:00.000Z");
const { companyId, runId } = await seedRunningRun({
now,
ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000,
});
const heartbeat = heartbeatService(db);
// First scan: creates the evaluation issue.
const first = await heartbeat.scanSilentActiveRuns({ now, companyId });
expect(first.created).toBe(1);
const evaluationIssueId = first.evaluationIssueIds[0];
expect(evaluationIssueId).toBeTruthy();
// Reviewer closes the issue on the board — no watchdog decision recorded.
await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId));
// Second scan: should NOT create a new issue. Instead auto-records dismissed_false_positive.
const second = await heartbeat.scanSilentActiveRuns({ now, companyId });
expect(second.created).toBe(0);
expect(second.skipped).toBe(1);
// All subsequent scans also skip.
const third = await heartbeat.scanSilentActiveRuns({ now, companyId });
expect(third.created).toBe(0);
expect(third.skipped).toBe(1);
// The auto-recorded dismissed_false_positive decision must be persisted.
const decisions = await db
.select()
.from(heartbeatRunWatchdogDecisions)
.where(and(eq(heartbeatRunWatchdogDecisions.runId, runId), eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive")));
expect(decisions).toHaveLength(1);
expect(decisions[0].evaluationIssueId).toBe(evaluationIssueId);
});
it("still allows re-arm after continue decision even when issue was closed on board", async () => {
// When a human explicitly recorded a 'continue' decision the watchdog lifecycle is active;
// closing the issue on the board should not permanently suppress future alerts.
const now = new Date("2026-04-22T20:00:00.000Z");
const { companyId, managerId, runId } = await seedRunningRun({
now,
ageMs: ACTIVE_RUN_OUTPUT_SUSPICION_THRESHOLD_MS + 60_000,
});
const heartbeat = heartbeatService(db);
const recovery = recoveryService(db, { enqueueWakeup: vi.fn() });
const scan = await heartbeat.scanSilentActiveRuns({ now, companyId });
const evaluationIssueId = scan.evaluationIssueIds[0];
expect(evaluationIssueId).toBeTruthy();
// Human records a 'continue' decision.
await recovery.recordWatchdogDecision({
runId,
actor: { type: "agent", agentId: managerId },
decision: "continue",
evaluationIssueId,
reason: "Expected quiet period.",
now,
});
// Board also closes the evaluation issue.
await db.update(issues).set({ status: "done" }).where(eq(issues.id, evaluationIssueId));
// After the re-arm window, watchdog should still fire a new alert.
const rearmAt = new Date(now.getTime() + ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS + 60_000);
const rearm = await heartbeat.scanSilentActiveRuns({ now: rearmAt, companyId });
expect(rearm.created).toBe(1);
expect(rearm.evaluationIssueIds[0]).not.toBe(evaluationIssueId);
});
it("validates createdByRunId before storing watchdog decisions", async () => { it("validates createdByRunId before storing watchdog decisions", async () => {
const now = new Date("2026-04-22T20:00:00.000Z"); const now = new Date("2026-04-22T20:00:00.000Z");
const { companyId, managerId, runId } = await seedRunningRun({ const { companyId, managerId, runId } = await seedRunningRun({
+132 -13
View File
@@ -898,6 +898,51 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
return row ?? null; return row ?? null;
} }
// Returns a `done` stale-run evaluation issue for this run if one exists.
// Used to detect when a reviewer closed an alert directly on the board without going through
// the watchdog decision API — which would not leave a dismissed_false_positive decision record.
//
// Scoped to `done` only (not `cancelled`): cancellation is used by other system code paths
// and does not imply a reviewer's "false positive" verdict. `done` is the explicit
// board-close path used by reviewers acknowledging the alert. A cancelled evaluation is
// allowed to re-fire on the next scan; if a reviewer wants permanent suppression they
// should mark the alert done or record a watchdog decision.
async function findClosedStaleRunEvaluation(companyId: string, runId: string) {
const [row] = await db
.select({ id: issues.id, identifier: issues.identifier, status: issues.status })
.from(issues)
.where(
and(
eq(issues.companyId, companyId),
eq(issues.originKind, STALE_ACTIVE_RUN_EVALUATION_ORIGIN_KIND),
eq(issues.originId, runId),
isNull(issues.hiddenAt),
eq(issues.status, "done"),
),
)
.orderBy(desc(issues.updatedAt))
.limit(1);
return row ?? null;
}
// Returns true when a reviewer has already dismissed this run's silence as a false positive.
// Used to prevent re-filing after a deliberate close — while still allowing legitimate
// re-arm after a "continue" decision's snooze window expires.
async function hasDismissedFalsePositiveDecision(companyId: string, runId: string) {
const [row] = await db
.select({ id: heartbeatRunWatchdogDecisions.id })
.from(heartbeatRunWatchdogDecisions)
.where(
and(
eq(heartbeatRunWatchdogDecisions.companyId, companyId),
eq(heartbeatRunWatchdogDecisions.runId, runId),
eq(heartbeatRunWatchdogDecisions.decision, "dismissed_false_positive"),
),
)
.limit(1);
return row != null;
}
async function buildRunOutputSilence( async function buildRunOutputSilence(
run: Pick< run: Pick<
typeof heartbeatRuns.$inferSelect, typeof heartbeatRuns.$inferSelect,
@@ -1462,26 +1507,42 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
); );
} }
async function ensureSourceIssueBlockedByStaleEvaluation(input: { async function ensureSourceIssueCommentedForStaleEvaluation(input: {
sourceIssue: typeof issues.$inferSelect | null; sourceIssue: typeof issues.$inferSelect | null;
evaluationIssue: { id: string; identifier: string | null }; evaluationIssue: { id: string; identifier: string | null };
run: typeof heartbeatRuns.$inferSelect; run: typeof heartbeatRuns.$inferSelect;
}) { }) {
if (!input.sourceIssue || ["done", "cancelled"].includes(input.sourceIssue.status)) return false; if (!input.sourceIssue || ["done", "cancelled"].includes(input.sourceIssue.status)) return false;
const blockerIds = await existingBlockerIssueIds(input.sourceIssue.companyId, input.sourceIssue.id); // Idempotency guard: if we've already emitted the escalation comment for this
if (blockerIds.includes(input.evaluationIssue.id)) return false; // (sourceIssue, evaluationIssue) pair, skip. Without this, every subsequent scan
const nextBlockerIds = [...blockerIds, input.evaluationIssue.id]; // cycle while the evaluation issue is still open re-fires the comment and spams
await issuesSvc.update(input.sourceIssue.id, { // the source-issue thread. The activity log row written below is the persistence
...(input.sourceIssue.status === "blocked" ? {} : { status: "blocked" }), // record we check against — a single row per pair is enough to suppress repeats
blockedByIssueIds: nextBlockerIds, // even after process restarts.
}); const [priorEscalation] = await db
.select({ id: activityLog.id })
.from(activityLog)
.where(
and(
eq(activityLog.companyId, input.sourceIssue.companyId),
eq(activityLog.action, "heartbeat.output_stale_escalated"),
eq(activityLog.entityType, "issue"),
eq(activityLog.entityId, input.sourceIssue.id),
sql`${activityLog.details} ->> 'evaluationIssueId' = ${input.evaluationIssue.id}`,
),
)
.limit(1);
if (priorEscalation) return false;
// Evaluation issues are observability-only — do NOT add them to blockedByIssueIds.
// They are already parented under the source issue. Adding them as hard blockers
// creates a self-amplifying loop: block → silence → new alert → block again.
await issuesSvc.addComment(input.sourceIssue.id, [ await issuesSvc.addComment(input.sourceIssue.id, [
"Paperclip detected critical output silence on this issue's active run.", "Paperclip detected critical output silence on this issue's active run.",
"", "",
`- Evaluation issue: ${input.evaluationIssue.identifier ?? input.evaluationIssue.id}`, `- Evaluation issue: ${input.evaluationIssue.identifier ?? input.evaluationIssue.id}`,
`- Run: \`${input.run.id}\``, `- Run: \`${input.run.id}\``,
"", "",
"This blocks the source issue on the explicit review task without cancelling the active process.", "Review the evaluation issue above. The active run has not been cancelled.",
].join("\n"), { runId: input.run.id }); ].join("\n"), { runId: input.run.id });
await logActivity(db, { await logActivity(db, {
companyId: input.sourceIssue.companyId, companyId: input.sourceIssue.companyId,
@@ -1495,7 +1556,6 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
details: { details: {
source: "recovery.scan_silent_active_runs", source: "recovery.scan_silent_active_runs",
evaluationIssueId: input.evaluationIssue.id, evaluationIssueId: input.evaluationIssue.id,
blockerIssueIds: nextBlockerIds,
}, },
}); });
return true; return true;
@@ -1549,6 +1609,65 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
}); });
} }
} }
// Idle output is expected when the source issue is blocked — skip ticket creation entirely.
if (sourceIssue?.status === "blocked") return { kind: "skipped" as const };
// Dedup: if a reviewer has dismissed this run's silence as a false positive, don't re-file.
// A "continue" decision with a snooze window is allowed to re-arm normally — only an
// explicit dismissed_false_positive blocks all further alerts for this run.
if (await hasDismissedFalsePositiveDecision(input.run.companyId, input.run.id)) {
return { kind: "skipped" as const };
}
// Dedup: if a prior evaluation issue for this run was closed `done` on the board
// without going through the watchdog decision API, no dismissed_false_positive record exists
// and the watchdog would re-fire every cycle. Auto-record the suppression now so future
// cycles skip immediately via hasDismissedFalsePositiveDecision.
//
// Exception: if any watchdog decision exists (snooze/continue), a human explicitly opted
// in to the watchdog lifecycle — honour that and allow re-arm as designed.
//
// Concurrency: the check-then-insert runs inside a transaction with a per-(company,run)
// advisory lock so two overlapping scans cannot both observe `hasAnyDecision = false`
// and both insert a dismissed_false_positive row. The table has no unique constraint
// on (companyId, runId, decision), so the advisory lock is the serialization point.
const closedEvaluation = await findClosedStaleRunEvaluation(input.run.companyId, input.run.id);
if (closedEvaluation) {
const autoDismissed = await db.transaction(async (tx) => {
await tx.execute(
sql`SELECT pg_advisory_xact_lock(hashtextextended(${`watchdog_dismiss:${input.run.companyId}:${input.run.id}`}, 0))`,
);
const hasAnyDecision = await tx
.select({ id: heartbeatRunWatchdogDecisions.id })
.from(heartbeatRunWatchdogDecisions)
.where(
and(
eq(heartbeatRunWatchdogDecisions.companyId, input.run.companyId),
eq(heartbeatRunWatchdogDecisions.runId, input.run.id),
),
)
.limit(1)
.then((rows) => rows.length > 0);
if (hasAnyDecision) return false;
await tx.insert(heartbeatRunWatchdogDecisions).values({
companyId: input.run.companyId,
runId: input.run.id,
evaluationIssueId: closedEvaluation.id,
decision: "dismissed_false_positive",
snoozedUntil: null,
reason: `Auto-recorded: evaluation issue ${closedEvaluation.identifier} was closed as ${closedEvaluation.status} on the board without a watchdog decision.`,
createdByAgentId: null,
createdByUserId: null,
createdByRunId: null,
});
return true;
});
if (autoDismissed) {
return { kind: "skipped" as const };
}
}
const prefix = await getCompanyIssuePrefix(input.run.companyId); const prefix = await getCompanyIssuePrefix(input.run.companyId);
const evidence = await collectStaleRunEvidence({ const evidence = await collectStaleRunEvidence({
run: input.run, run: input.run,
@@ -1570,7 +1689,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
`- Silent for: ${formatDuration(evidence.silenceAgeMs)}`, `- Silent for: ${formatDuration(evidence.silenceAgeMs)}`,
`- Last output at: ${input.run.lastOutputAt?.toISOString() ?? "none recorded"}`, `- Last output at: ${input.run.lastOutputAt?.toISOString() ?? "none recorded"}`,
].join("\n"), { runId: input.run.id }); ].join("\n"), { runId: input.run.id });
await ensureSourceIssueBlockedByStaleEvaluation({ await ensureSourceIssueCommentedForStaleEvaluation({
sourceIssue, sourceIssue,
evaluationIssue: existing, evaluationIssue: existing,
run: input.run, run: input.run,
@@ -1578,7 +1697,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
return { kind: "escalated" as const, evaluationIssueId: existing.id }; return { kind: "escalated" as const, evaluationIssueId: existing.id };
} }
if (level === "critical") { if (level === "critical") {
await ensureSourceIssueBlockedByStaleEvaluation({ await ensureSourceIssueCommentedForStaleEvaluation({
sourceIssue, sourceIssue,
evaluationIssue: existing, evaluationIssue: existing,
run: input.run, run: input.run,
@@ -1640,7 +1759,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
}, },
}); });
if (level === "critical") { if (level === "critical") {
await ensureSourceIssueBlockedByStaleEvaluation({ await ensureSourceIssueCommentedForStaleEvaluation({
sourceIssue, sourceIssue,
evaluationIssue: evaluation, evaluationIssue: evaluation,
run: input.run, run: input.run,