Files
paperclip/server
sunghere 130219c0be fix(recovery): exempt stranded escalation when assignee shows recent visible progress (#5213)
## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - The recovery service watches `in_progress` agent-assigned issues
every 30s and creates "Recover stalled issue …" child issues when
execution looks stranded
> - The `isRepeatedProductiveContinuationRecovery` branch escalates
after just **two consecutive productive continuation runs** — fine for
genuinely stuck agents that loop without doing anything, but a false
positive for batch workflows that legitimately advance every heartbeat
(e.g. multi-frame image generation that produces 1–2 frames + an
attachment per heartbeat)
> - In production this fired ~95 times for a 19-character batch run,
burning a recovery owner heartbeat each time
> - This pull request adds a "recent visible progress" exemption: if the
assignee posted a comment or any attachment within the exemption window
(default 30 min, env-tunable, 60s floor), skip the escalation and let
the normal continuation-retry path enqueue the next wake
> - The benefit is one platform tweak unblocks all current and future
batch workflows without weakening the genuinely-stuck case — agents that
go silent still escalate after the window elapses

## What Changed

- `server/src/services/recovery/service.ts`
- new `STRANDED_RECENT_PROGRESS_EXEMPTION_MS` constant (default 30 min,
override via env, floored at 60s)
- new `hasRecentVisibleProgress(companyId, issueId, assigneeAgentId,
windowMs)` helper — single parallel query against `issue_comments`
(filtered by `authorAgentId`) + `issue_attachments`, both using existing
indexes
- in `reconcileStrandedAssignedIssues`, the
`isRepeatedProductiveContinuationRecovery` branch now consults the
helper before escalating; on exemption it falls through to the existing
continuation-retry enqueue path
- new `recentProgressExempted` counter on the reconcile result, surfaced
in the periodic recovery log via the existing `...reconciled` spread
- `server/src/__tests__/heartbeat-process-recovery.test.ts`
- new test: recent agent comment → no escalation, continuation
re-queued, `recentProgressExempted: 1`
- new test: stale (24h-old) agent comment → escalation still fires as
before

## Verification

- `pnpm typecheck` — green across the workspace
- `pnpm exec vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts` — 39/39 pass
(37 pre-existing + 2 new)
- Smoke after deploy: confirm Image Spec multi-frame generation no
longer creates `Recover stalled issue …` child issues per heartbeat

## Risks

- **Behavioral shift, low blast radius.** A genuinely-stuck agent that
posts cosmetic comments every <30 min would now escalate later instead
of immediately. Mitigated by:
  - Window is configurable via `STRANDED_RECENT_PROGRESS_EXEMPTION_MS`
- Other escalation paths are untouched (failed/cancelled/timed_out runs
still escalate immediately, paused-tree handling unchanged,
recovery-issue-on-recovery guard unchanged)
- Periodic recovery log now reports `recentProgressExempted` so a
runaway exemption is visible in operations
- No DB migration required — both `issue_comments` and
`issue_attachments` queries use existing indexes
- Backward compatible: pre-existing test "blocks stranded in-progress
work after a productive continuation retry was already used" still
passes unchanged because no comment is seeded → no exemption → escalates

## Model Used

- Claude Opus 4.7 (`claude-opus-4-7`), extended thinking, tool use
enabled. Investigation, change, tests, and PR body all human-supervised
through a Paperclip agent heartbeat.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots — N/A, server-only
- [ ] I have updated relevant documentation to reflect my changes — no
docs touched the previous behavior; the env knob is self-documenting via
the comment in `service.ts`
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge


## Cross-references and status (maintainer)

Rebased onto current `master`. No duplicate PRs absorbed.

Refs #6072 — related open report in the same
`reconcileStrandedAssignedIssues` /
`isRepeatedProductiveContinuationRecovery` family (stale
productive-continuation evidence). This PR does not fix #6072, but the
recent-visible-progress exemption added here shrinks the false-positive
surface in that branch and the new `recentProgressExempted` counter
gives operators visibility into the broader escalation path.

Co-authored-by: sunghere <sunghere@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-11 21:53:51 -07:00
..