> **Note (rebase, 2026-06-11):** this PR was rebased onto current
`master` again after #6008 (`Clear stale checkoutRunId on run
finalization and add backstop sweeper`) landed. See "What Changed" below
for how the previous narrow per-issue checkoutRunId clear from #6008 is
now subsumed by a single bulk-update pass over every sibling that still
references the finalizing run, with the two columns cleared in separate,
scoped UPDATEs so retry pointers are not clobbered.
## Thinking Path
> - Paperclip orchestrates AI-agent companies; issue execution ownership
is gated by `executionRunId` / `executionAgentNameKey` /
`executionLockedAt`, and any checkout whose run doesn't match the stored
`executionRunId` is rejected with 409 "Issue run ownership conflict"
> - In production, a running company silently got stuck: multiple
in-progress issues ended up with `executionRunId` pointing at heartbeat
runs that had already finalized hours earlier, so every new agent
checkout returned 409 and the issues stayed marked blocked forever
> - Root cause: `releaseIssueExecutionAndPromote` only resolved and
cleared the execution lock on one issue per finalizing run (the run's
`contextSnapshot.issueId`, or `rows[0]` when no context issue existed),
but `enqueueWakeup`'s "legacy run" fallback can stamp the same `run.id`
onto sibling issues' `execution_run_id`, so the siblings were left as
orphans
> - #4258 shipped a *reactive* fix for this bug class in `issueService`:
`clearExecutionRunIfTerminal` now self-heals a stale execution lock on
the next ownership-gated access (`checkout`, `assertCheckoutOwner`,
`release`) to each affected issue, and `release` now unconditionally
clears the three execution-lock fields
> - #6008 shipped the *symmetric* fix for the `checkoutRunId` column
(per-issue self-heal in `releaseIssueExecutionAndPromote`,
`clearCheckoutRunIfTerminal` helper, and a backstop sweeper)
> - This PR adds the *proactive* half at the point of run finalization,
and generalizes #6008's per-issue checkoutRunId clear to every sibling
that still references the finalizing run. After this PR + #4258 + #6008,
orphan locks are cleared at the moment the run ends (across both
execution and checkout columns, on every affected sibling), not only on
the next access attempt to one of them
## Linked Issues or Issue Description
- Closes#4194
- Closes#201
- Closes#3904
## What Changed
- **`server/src/services/heartbeat.ts` —
`releaseIssueExecutionAndPromote`:** lock the context issue (when set)
**and** every issue still referencing the finalizing run via either
`execution_run_id` or `checkout_run_id`, under a single `SELECT ... FOR
UPDATE ORDER BY id` (deterministic lock acquisition across concurrent
finalizations). Then issue two scoped bulk `UPDATE`s in the same
transaction:
- one clears `executionRunId` / `executionAgentNameKey` /
`executionLockedAt` on every issue whose `executionRunId` still matches
this run,
- the other clears `checkoutRunId` on every issue whose `checkoutRunId`
still matches this run.
The split avoids clobbering a retry's `executionRunId` pointer: in the
codex-transient-upstream and process-loss retry paths, `executionRunId`
is moved from this run to the retry run before
`releaseIssueExecutionAndPromote` runs, while `checkoutRunId` is left
pinned at the failed run. A single combined `UPDATE` with an `OR`
predicate would null the retry's `executionRunId` in that case — these
two scoped UPDATEs do not.
The deferred-wake promotion contract is preserved: pick the run's
context issue when present, else the first candidate (matching the
legacy `rows[0]` selection under the new ordering). Recovery-agent
fields added by a concurrent master change (`taskKey`, `recoveryAgent`,
`recoverySessionBefore`, `recoveryAgentNameKey`, and the extra
`assigneeAgentId`/`assigneeUserId` columns used downstream for
`issueNeedsImmediateRecovery`) are fully preserved through the merge.
The workspace-validation-failed recovery-comment path added by master is
also preserved on the primary issue.
- **`server/src/__tests__/execution-lock-orphan-cleanup.test.ts` (new, 6
tests):** multi-issue cleanup on finalize (2 issues); higher fan-out (4
issues) exercising the bulk `UPDATE` path; finalization of a run without
a `contextSnapshot.issueId`; cross-company isolation under a
pathologically cross-tenant `executionRunId`; unrelated-run locks are
never touched by a sibling run's finalization; and a dedicated test for
the `checkoutRunId` bulk-clear path that proves the split-UPDATE
invariant by seeding a sibling whose `executionRunId` already points at
a retry run while `checkoutRunId` is still pinned at the finalizing run
— the test asserts the retry pointer is preserved and the checkout
column is cleared.
**Not in this PR:** `server/src/services/issues.ts` is intentionally
unchanged. The release-side changes from the previous revision of this
PR are fully subsumed by #4258 (`clearExecutionRunIfTerminal` plus
unconditional clear in `release`) and #6008
(`clearCheckoutRunIfTerminal`). The per-issue checkoutRunId clear added
in `releaseIssueExecutionAndPromote` by #6008 is replaced by the bulk
path here, which strictly widens coverage from "primary issue only" to
"every sibling that still references this run".
## Verification
- `pnpm install --frozen-lockfile` — clean
- `pnpm typecheck` (server workspace) — passes on the rebased branch
- Focused suite (8 files, 147 tests — `execution-lock-orphan-cleanup`
(6), `heartbeat-run-log`, `heartbeat-run-summary`,
`issues-checkout-wakeup`, `issue-execution-policy-routes`,
`issue-agent-mutation-ownership-routes`, `issues-service`, and
`issue-stale-execution-lock-routes`): **147/147 pass**.
- `heartbeat-process-recovery.test.ts` (52 tests): 51 pass; the one
failure (`queues exactly one retry when the recorded local pid is dead`)
reproduces verbatim on raw `master` with the PR's changes reverted, so
it is a pre-existing flake (also noted by the earlier CI-retrigger
commit on this branch).
- Regression evidence: reverting `server/src/services/heartbeat.ts` to
`master` while keeping the 6 new tests causes 5 of them to fail (the
finalization-cleanup tests, including the new
checkoutRunId-pointer-preservation test; the "unrelated-run locks never
touched" test passes either way — that's its purpose as a negative
control); restoring the fix returns to 6/6 green.
- `pnpm-lock.yaml` untouched; no migration required; no public API shape
change.
Repro-ability of the original production symptom:
```
# Seed an issue with executionRunId pointing at a finalized run
# (matches what enqueueWakeup's legacy-run fallback can produce)
UPDATE issues SET execution_run_id = '<finalized-run-id>',
execution_agent_name_key = 'ceo',
execution_locked_at = NOW()
WHERE id = '<issue-id>';
# Any subsequent svc.checkout against this issue 409s until the
# lock is cleared. Before #4258, the lock stayed forever. After
# #4258, it self-heals on next ownership-gated access. After this
# PR, it's cleared at the moment the run finalizes so an untouched
# sibling issue doesn't rely on a later access to recover.
```
## Risks
- **Same bug class exists in two adjacent, untouched code paths in this
file** — `enqueueProcessLossRetry` repoints `executionRunId` only for
the context issue (not siblings stamped with the failed run id), and
`enqueueMissingIssueCommentRetry` locks all matching issues under `FOR
UPDATE` but only updates the first row returned. Filed separately as
#4319 with exact line refs so this PR can stay narrow.
- **No `activity_log` entry is emitted for secondary orphan issues**
whose locks are cleared by the new bulk UPDATEs — only the single
primary issue retains its existing promotion event stream. Callers that
audit lock transitions purely via `activity_log` may see orphan issues
flip to `execution_run_id = null` (or `checkout_run_id = null`) without
a matching event. Easy to add a batched log line in a follow-up if audit
completeness matters; the #6008 backstop sweeper already emits
`issue.stale_lock_cleared` for the catch-up path so this is mainly an
observability nicety for the proactive path.
- **UI polling shift** — `ui/src/pages/IssueDetail.tsx` and
`ui/src/lib/issueActiveRun.ts` key off `execution_run_id` for active-run
polling; clearing orphan locks at finalize (rather than waiting for next
access as in #4258's flow) means the "executing" UI state falls back
slightly faster when the underlying run has genuinely finalized.
Observable, but an improvement over showing stuck state.
- **Lockfile and manifests untouched**; no migration required; no public
API shape change.
## Model Used
- **Provider/model:** Anthropic Claude Opus 4.7 (the model this session
is running on, per Cursor IDE system context)
- **Harness:** Cursor IDE
- **Capabilities used:** extended/reasoning thinking mode; filesystem
and shell tool use; parallel subagent orchestration for a three-lens
readonly self-review (correctness+concurrency, regression blast radius,
test adequacy+style) before the initial push; targeted rebase conflict
resolution after #4258 landed and again after #6008 landed, combining
all three branches' changes to `releaseIssueExecutionAndPromote`
- **Context:** full repository plus live access to the running Paperclip
instance that exhibited the bug (the instance was unblocked via a
targeted DB intervention before this code fix was authored; the live
observation drove the root-cause analysis)
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work (the roadmap contains no references to heartbeat
execution-lock management or `releaseIssueExecutionAndPromote`)
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#4258 and #6008 are the closest prior art and are explicitly
cross-linked in the thinking path and "What Changed" sections; no other
open or merged PR touches `releaseIssueExecutionAndPromote`)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template — see "Linked Issues or Issue Description" above
- [x] I have run tests locally and they pass (typecheck workspace-wide;
147/147 in the focused suite including #4258's and #6008's new tests)
- [x] I have added or updated tests where applicable (6 regression
tests; 5 of them provably fail on `master` without the code change)
- [ ] If this change affects the UI, I have included before/after
screenshots — *n/a, this is a server-only change; any UI polling effect
is documented under Risks*
- [ ] I have updated relevant documentation to reflect my changes —
*n/a, no user-facing or API docs reference
`releaseIssueExecutionAndPromote`*
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green (the prior `verify` flake is
unrelated to this PR's change path and reproduces on unrelated branches;
documented above and in follow-up #4328)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(the only test-coverage gap Greptile flagged on the latest review — the
`checkoutRunId` bulk-clear branch — is now covered by the new 6th test
in this revision)
- [x] I will address all Greptile and reviewer comments before
requesting merge
Co-authored-by: Devin Foley <devin@paperclip.ing>
Co-authored-by: Paperclip <noreply@paperclip.ing>
AyeletMorris-ShieldFC
d2ef767712