fix(server): adopt stale checkout run ownership (#5413)
## Thinking Path > - Paperclip is a control plane for autonomous AI-agent companies. > - Issue checkout ownership is part of the execution-control layer that prevents two runs from mutating the same task at the same time. > - The current lock model should preserve `409` conflicts for live competing owners, but it should not strand the rightful assignee behind a stale terminal run. > - A same-agent follow-up run can encounter an existing `checkoutRunId` from a failed, timed-out, succeeded, or missing heartbeat run. > - In that case, the new run should safely adopt ownership instead of failing with an ownership conflict. > - This pull request makes stale checkout adoption transactional and keeps live checkout owners protected. > - The benefit is safer run recovery without weakening single-owner checkout semantics. ## Linked Issues or Issue Description - Fixes #5350 - Closes #1508 - Closes #1970 - Closes #2083 - Closes #3158 - Closes #3190 - Related stale-lock PRs reviewed during dedup search: #7536, #6658, #5660, #5442, #6223, #7048, #6824, #6799 ## What Changed - Updated issue checkout ownership recovery so the current assignee can adopt a stale terminal or missing checkout run. - Added row locking around stale checkout adoption to avoid races while replacing `checkoutRunId` / `executionRunId`. - Preserved `409` behavior when a different live checkout owner is still active. - Prevented terminal actor runs from reclaiming an unowned checkout lock after the newer eager stale-checkout clear path. - Fixed the stale checkout test fixture so same-assignee cases do not insert duplicate agent rows. - Added/kept focused coverage for stale checkout adoption and live-owner conflict behavior. - Fixes #5350. ## Verification - Focused tests: ```sh pnpm exec vitest run server/src/__tests__/issues-service.test.ts server/src/__tests__/issue-stale-execution-lock-routes.test.ts ``` Result: ```text 2 passed, 84 tests passed ``` - Server typecheck: ```sh pnpm --filter @paperclipai/server typecheck ``` Result: ```text passed ``` - Live curl smoke confirmed same-agent stale checkout adoption returns `200` instead of `409`. ```text old_run_status=succeeded checkout_http=200 patch_http=200 ``` The PATCH response showed `checkoutRunId` and `executionRunId` updated to the new run id. ### Live curl smoke result <img width="1498" height="570" alt="Live curl smoke showing stale checkout adoption returned 200" src="https://github.com/user-attachments/assets/4bf834de-e3cd-4495-ac5a-74767b439eeb" /> ### Server request log <img width="631" height="131" alt="Server logs showing heartbeat, checkout, and patch requests succeeded" src="https://github.com/user-attachments/assets/ceaaa403-110e-44e8-bac8-5d8506e79cc3" /> ## Risks - Low to medium risk: this touches issue execution lock ownership. - The behavioral shift is intentionally narrow: only the current assignee can adopt stale terminal or missing checkout ownership. - Live checkout owners remain protected with `409`. - No database migration or API contract change. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI GPT-5.5 Codex coding agent with repository tool use, shell execution, code review, and local verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge ## Cross-references and status (maintainer) - Closes #1508 - Closes #1970 - Closes #2083 - Closes #3158 - Closes #3190 - Status: rebased onto current master; focused tests and server typecheck pass locally; all required CI is green; Greptile is 5/5; master drift verified. --------- Co-authored-by: Devin Foley <devin@paperclip.ing>
This commit is contained in:
Vasu Yadav
+257
-117
@@ -3661,36 +3661,99 @@ export function issueService(db: Db) {
|
||||
actorRunId: string;
|
||||
expectedCheckoutRunId: string;
|
||||
}) {
|
||||
const stale = await isTerminalOrMissingHeartbeatRun(input.expectedCheckoutRunId);
|
||||
if (!stale) return null;
|
||||
return db.transaction(async (tx) => {
|
||||
const lockedIssue = await tx
|
||||
.select({
|
||||
id: issues.id,
|
||||
status: issues.status,
|
||||
assigneeAgentId: issues.assigneeAgentId,
|
||||
checkoutRunId: issues.checkoutRunId,
|
||||
executionRunId: issues.executionRunId,
|
||||
})
|
||||
.from(issues)
|
||||
.where(eq(issues.id, input.issueId))
|
||||
.for("update")
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!lockedIssue) {
|
||||
return { adopted: null, latest: null };
|
||||
}
|
||||
|
||||
const now = new Date();
|
||||
const adopted = await db
|
||||
.update(issues)
|
||||
.set({
|
||||
checkoutRunId: input.actorRunId,
|
||||
executionRunId: input.actorRunId,
|
||||
executionLockedAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(
|
||||
and(
|
||||
eq(issues.id, input.issueId),
|
||||
eq(issues.status, "in_progress"),
|
||||
eq(issues.assigneeAgentId, input.actorAgentId),
|
||||
eq(issues.checkoutRunId, input.expectedCheckoutRunId),
|
||||
if (
|
||||
lockedIssue.status !== "in_progress" ||
|
||||
lockedIssue.assigneeAgentId !== input.actorAgentId ||
|
||||
lockedIssue.checkoutRunId !== input.expectedCheckoutRunId
|
||||
) {
|
||||
return { adopted: null, latest: lockedIssue };
|
||||
}
|
||||
|
||||
await Promise.all([
|
||||
tx.execute(
|
||||
sql`select ${heartbeatRuns.id} from ${heartbeatRuns} where ${heartbeatRuns.id} = ${input.expectedCheckoutRunId} for update`,
|
||||
),
|
||||
)
|
||||
.returning({
|
||||
id: issues.id,
|
||||
status: issues.status,
|
||||
assigneeAgentId: issues.assigneeAgentId,
|
||||
checkoutRunId: issues.checkoutRunId,
|
||||
executionRunId: issues.executionRunId,
|
||||
})
|
||||
.then((rows) => rows[0] ?? null);
|
||||
tx.execute(
|
||||
sql`select ${heartbeatRuns.id} from ${heartbeatRuns} where ${heartbeatRuns.id} = ${input.actorRunId} for update`,
|
||||
),
|
||||
]);
|
||||
const [existingRun, actorRun] = await Promise.all([
|
||||
tx
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, input.expectedCheckoutRunId))
|
||||
.then((rows) => rows[0] ?? null),
|
||||
tx
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, input.actorRunId))
|
||||
.then((rows) => rows[0] ?? null),
|
||||
]);
|
||||
const stale = !existingRun || TERMINAL_HEARTBEAT_RUN_STATUSES.has(existingRun.status);
|
||||
const actorLive = actorRun && !TERMINAL_HEARTBEAT_RUN_STATUSES.has(actorRun.status);
|
||||
if (!stale || !actorLive) {
|
||||
return { adopted: null, latest: lockedIssue };
|
||||
}
|
||||
|
||||
return adopted;
|
||||
const now = new Date();
|
||||
const adopted = await tx
|
||||
.update(issues)
|
||||
.set({
|
||||
checkoutRunId: input.actorRunId,
|
||||
executionRunId: input.actorRunId,
|
||||
executionLockedAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(
|
||||
and(
|
||||
eq(issues.id, input.issueId),
|
||||
eq(issues.status, "in_progress"),
|
||||
eq(issues.assigneeAgentId, input.actorAgentId),
|
||||
eq(issues.checkoutRunId, input.expectedCheckoutRunId),
|
||||
),
|
||||
)
|
||||
.returning({
|
||||
id: issues.id,
|
||||
status: issues.status,
|
||||
assigneeAgentId: issues.assigneeAgentId,
|
||||
checkoutRunId: issues.checkoutRunId,
|
||||
executionRunId: issues.executionRunId,
|
||||
})
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (adopted) {
|
||||
return { adopted, latest: adopted };
|
||||
}
|
||||
|
||||
const latest = await tx
|
||||
.select({
|
||||
id: issues.id,
|
||||
status: issues.status,
|
||||
assigneeAgentId: issues.assigneeAgentId,
|
||||
checkoutRunId: issues.checkoutRunId,
|
||||
executionRunId: issues.executionRunId,
|
||||
})
|
||||
.from(issues)
|
||||
.where(eq(issues.id, input.issueId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
return { adopted: null, latest };
|
||||
});
|
||||
}
|
||||
|
||||
async function adoptUnownedCheckoutRun(input: {
|
||||
@@ -3698,34 +3761,46 @@ export function issueService(db: Db) {
|
||||
actorAgentId: string;
|
||||
actorRunId: string;
|
||||
}) {
|
||||
const now = new Date();
|
||||
const adopted = await db
|
||||
.update(issues)
|
||||
.set({
|
||||
checkoutRunId: input.actorRunId,
|
||||
executionRunId: input.actorRunId,
|
||||
executionLockedAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(
|
||||
and(
|
||||
eq(issues.id, input.issueId),
|
||||
eq(issues.status, "in_progress"),
|
||||
eq(issues.assigneeAgentId, input.actorAgentId),
|
||||
isNull(issues.checkoutRunId),
|
||||
or(isNull(issues.executionRunId), eq(issues.executionRunId, input.actorRunId)),
|
||||
),
|
||||
)
|
||||
.returning({
|
||||
id: issues.id,
|
||||
status: issues.status,
|
||||
assigneeAgentId: issues.assigneeAgentId,
|
||||
checkoutRunId: issues.checkoutRunId,
|
||||
executionRunId: issues.executionRunId,
|
||||
})
|
||||
.then((rows) => rows[0] ?? null);
|
||||
return db.transaction(async (tx) => {
|
||||
await tx.execute(
|
||||
sql`select ${heartbeatRuns.id} from ${heartbeatRuns} where ${heartbeatRuns.id} = ${input.actorRunId} for update`,
|
||||
);
|
||||
const actorRun = await tx
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, input.actorRunId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!actorRun || TERMINAL_HEARTBEAT_RUN_STATUSES.has(actorRun.status)) return null;
|
||||
|
||||
return adopted;
|
||||
const now = new Date();
|
||||
const adopted = await tx
|
||||
.update(issues)
|
||||
.set({
|
||||
checkoutRunId: input.actorRunId,
|
||||
executionRunId: input.actorRunId,
|
||||
executionLockedAt: now,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(
|
||||
and(
|
||||
eq(issues.id, input.issueId),
|
||||
eq(issues.status, "in_progress"),
|
||||
eq(issues.assigneeAgentId, input.actorAgentId),
|
||||
isNull(issues.checkoutRunId),
|
||||
or(isNull(issues.executionRunId), eq(issues.executionRunId, input.actorRunId)),
|
||||
),
|
||||
)
|
||||
.returning({
|
||||
id: issues.id,
|
||||
status: issues.status,
|
||||
assigneeAgentId: issues.assigneeAgentId,
|
||||
checkoutRunId: issues.checkoutRunId,
|
||||
executionRunId: issues.executionRunId,
|
||||
})
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
return adopted;
|
||||
});
|
||||
}
|
||||
|
||||
async function clearExecutionRunIfTerminal(issueId: string): Promise<boolean> {
|
||||
@@ -5480,13 +5555,13 @@ export function issueService(db: Db) {
|
||||
current.checkoutRunId &&
|
||||
current.checkoutRunId !== checkoutRunId
|
||||
) {
|
||||
const adopted = await adoptStaleCheckoutRun({
|
||||
const staleAdoption = await adoptStaleCheckoutRun({
|
||||
issueId: id,
|
||||
actorAgentId: agentId,
|
||||
actorRunId: checkoutRunId,
|
||||
expectedCheckoutRunId: current.checkoutRunId,
|
||||
});
|
||||
if (adopted) {
|
||||
if (staleAdoption.adopted) {
|
||||
const row = await db.select().from(issues).where(eq(issues.id, id)).then((rows) => rows[0] ?? null);
|
||||
if (!row) throw notFound("Issue not found");
|
||||
const [enriched] = await withIssueLabels(db, [row]);
|
||||
@@ -5562,77 +5637,142 @@ export function issueService(db: Db) {
|
||||
assertCheckoutOwner: async (id: string, actorAgentId: string, actorRunId: string | null) => {
|
||||
await clearExecutionRunIfTerminal(id);
|
||||
await clearCheckoutRunIfTerminal(id);
|
||||
const current = await db
|
||||
.select({
|
||||
id: issues.id,
|
||||
status: issues.status,
|
||||
assigneeAgentId: issues.assigneeAgentId,
|
||||
checkoutRunId: issues.checkoutRunId,
|
||||
executionRunId: issues.executionRunId,
|
||||
})
|
||||
.from(issues)
|
||||
.where(eq(issues.id, id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
const loadCurrent = () =>
|
||||
db
|
||||
.select({
|
||||
id: issues.id,
|
||||
status: issues.status,
|
||||
assigneeAgentId: issues.assigneeAgentId,
|
||||
checkoutRunId: issues.checkoutRunId,
|
||||
executionRunId: issues.executionRunId,
|
||||
})
|
||||
.from(issues)
|
||||
.where(eq(issues.id, id))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
const current = await loadCurrent();
|
||||
|
||||
if (!current) throw notFound("Issue not found");
|
||||
|
||||
if (
|
||||
current.status === "in_progress" &&
|
||||
current.assigneeAgentId === actorAgentId &&
|
||||
sameRunLock(current.checkoutRunId, actorRunId)
|
||||
) {
|
||||
return { ...current, adoptedFromRunId: null as string | null };
|
||||
}
|
||||
const resolveSameRunOwnership = (candidate: {
|
||||
id: string;
|
||||
status: string;
|
||||
assigneeAgentId: string | null;
|
||||
checkoutRunId: string | null;
|
||||
executionRunId: string | null;
|
||||
}) => {
|
||||
if (
|
||||
candidate.status === "in_progress" &&
|
||||
candidate.assigneeAgentId === actorAgentId &&
|
||||
sameRunLock(candidate.checkoutRunId, actorRunId)
|
||||
) {
|
||||
return { ...candidate, adoptedFromRunId: null as string | null };
|
||||
}
|
||||
return null;
|
||||
};
|
||||
|
||||
if (
|
||||
actorRunId &&
|
||||
current.status === "in_progress" &&
|
||||
current.assigneeAgentId === actorAgentId &&
|
||||
current.checkoutRunId == null &&
|
||||
(current.executionRunId == null || current.executionRunId === actorRunId)
|
||||
) {
|
||||
const adopted = await adoptUnownedCheckoutRun({
|
||||
issueId: id,
|
||||
const canAdoptUnownedCheckout = (candidate: {
|
||||
status: string;
|
||||
assigneeAgentId: string | null;
|
||||
checkoutRunId: string | null;
|
||||
executionRunId: string | null;
|
||||
}) => (
|
||||
actorRunId
|
||||
&& candidate.status === "in_progress"
|
||||
&& candidate.assigneeAgentId === actorAgentId
|
||||
&& candidate.checkoutRunId == null
|
||||
&& (candidate.executionRunId == null || candidate.executionRunId === actorRunId)
|
||||
);
|
||||
|
||||
const resolveOwnership = async (
|
||||
candidate: {
|
||||
id: string;
|
||||
status: string;
|
||||
assigneeAgentId: string | null;
|
||||
checkoutRunId: string | null;
|
||||
executionRunId: string | null;
|
||||
},
|
||||
) => {
|
||||
const sameRunOwnership = resolveSameRunOwnership(candidate);
|
||||
if (sameRunOwnership) return { ownership: sameRunOwnership, latest: null };
|
||||
|
||||
if (canAdoptUnownedCheckout(candidate)) {
|
||||
const adopted = await adoptUnownedCheckoutRun({
|
||||
issueId: id,
|
||||
actorAgentId,
|
||||
actorRunId: actorRunId!,
|
||||
});
|
||||
|
||||
if (adopted) {
|
||||
return {
|
||||
ownership: {
|
||||
...adopted,
|
||||
adoptedFromRunId: null as string | null,
|
||||
},
|
||||
latest: null,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
actorRunId &&
|
||||
candidate.status === "in_progress" &&
|
||||
candidate.assigneeAgentId === actorAgentId &&
|
||||
candidate.checkoutRunId &&
|
||||
candidate.checkoutRunId !== actorRunId
|
||||
) {
|
||||
const previousCheckoutRunId = candidate.checkoutRunId;
|
||||
const staleAdoption = await adoptStaleCheckoutRun({
|
||||
issueId: id,
|
||||
actorAgentId,
|
||||
actorRunId,
|
||||
expectedCheckoutRunId: previousCheckoutRunId,
|
||||
});
|
||||
|
||||
if (staleAdoption.adopted) {
|
||||
return {
|
||||
ownership: {
|
||||
...staleAdoption.adopted,
|
||||
adoptedFromRunId: previousCheckoutRunId,
|
||||
},
|
||||
latest: null,
|
||||
};
|
||||
}
|
||||
|
||||
if (staleAdoption.latest) {
|
||||
const latestOwnership = resolveSameRunOwnership(staleAdoption.latest);
|
||||
if (latestOwnership) return { ownership: latestOwnership, latest: staleAdoption.latest };
|
||||
return { ownership: null, latest: staleAdoption.latest };
|
||||
}
|
||||
}
|
||||
|
||||
return { ownership: null, latest: null };
|
||||
};
|
||||
|
||||
const resolved = await resolveOwnership(current);
|
||||
if (resolved.ownership) return resolved.ownership;
|
||||
|
||||
const latest = resolved.latest ?? await loadCurrent();
|
||||
if (!latest) throw notFound("Issue not found");
|
||||
const resolvedLatest = await resolveOwnership(latest);
|
||||
if (resolvedLatest.ownership) return resolvedLatest.ownership;
|
||||
if (resolvedLatest.latest) {
|
||||
throw conflict("Issue run ownership conflict", {
|
||||
issueId: resolvedLatest.latest.id,
|
||||
status: resolvedLatest.latest.status,
|
||||
assigneeAgentId: resolvedLatest.latest.assigneeAgentId,
|
||||
checkoutRunId: resolvedLatest.latest.checkoutRunId,
|
||||
executionRunId: resolvedLatest.latest.executionRunId,
|
||||
actorAgentId,
|
||||
actorRunId,
|
||||
});
|
||||
|
||||
if (adopted) {
|
||||
return {
|
||||
...adopted,
|
||||
adoptedFromRunId: null as string | null,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
if (
|
||||
actorRunId &&
|
||||
current.status === "in_progress" &&
|
||||
current.assigneeAgentId === actorAgentId &&
|
||||
current.checkoutRunId &&
|
||||
current.checkoutRunId !== actorRunId
|
||||
) {
|
||||
const adopted = await adoptStaleCheckoutRun({
|
||||
issueId: id,
|
||||
actorAgentId,
|
||||
actorRunId,
|
||||
expectedCheckoutRunId: current.checkoutRunId,
|
||||
});
|
||||
|
||||
if (adopted) {
|
||||
return {
|
||||
...adopted,
|
||||
adoptedFromRunId: current.checkoutRunId,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
throw conflict("Issue run ownership conflict", {
|
||||
issueId: current.id,
|
||||
status: current.status,
|
||||
assigneeAgentId: current.assigneeAgentId,
|
||||
checkoutRunId: current.checkoutRunId,
|
||||
executionRunId: current.executionRunId,
|
||||
issueId: latest.id,
|
||||
status: latest.status,
|
||||
assigneeAgentId: latest.assigneeAgentId,
|
||||
checkoutRunId: latest.checkoutRunId,
|
||||
executionRunId: latest.executionRunId,
|
||||
actorAgentId,
|
||||
actorRunId,
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user