adapter-claude-local: recover from poisoned previous_message_id 400 (detect + clearSession) (#5972)
## Thinking Path
> - Paperclip's `claude_local` adapter persists Claude Code session
jsonls under `~/.claude/projects/…/{sessionId}.jsonl` and resumes them
on the next heartbeat
> - When Claude Code injects `<synthetic>` placeholder assistant
messages (after rate-limit, max-turn exhaustion, or transient-upstream
failures) those placeholders get UUID-format `message.id`s rather than
`msg_…`-format ids
> - On the next `--resume`, Claude Code passes that UUID as
`previous_message_id` and Anthropic's API rejects it with a 400:
``diagnostics.previous_message_id: must be the `id` from a prior
/v1/messages response (starts with `msg_`)``
> - The adapter had a session-rotation fallback only for "unknown
session" errors, so the poisoned session was `--resume`-d indefinitely
and the agent flipped between `idle` and `error` every heartbeat
> - Even worse, the *result* event of the failing run still carried a
`session_id`, and the adapter was persisting that id into the
issue-scoped session store (`agentTaskSessions`). So even after we
detected the 400, every subsequent continuation re-loaded the same
poisoned id and hit the same 400 again — the issue was permanently
stranded
> - We observed this on multiple agents in our deployment; the only
manual fix was to rename the `.jsonl`, which is not a viable long-term
workaround
> - This PR detects the 400, runs the same session-rotation fallback the
unknown-session path uses **and** stops persisting the poisoned id, so
the next attempt starts genuinely fresh
## Linked Issues or Issue Description
No external GitHub issue is linked. Describing the problem inline
following the bug-report template:
**What happened:** `claude_local` agents flipped between `idle` and
`error` on every heartbeat because the persisted session jsonl carried a
synthetic UUID `previous_message_id` (from `<synthetic>` assistant
placeholders injected after rate-limit/max-turn/upstream errors).
Anthropic's API rejected every `--resume` with a 400:
``diagnostics.previous_message_id: must be the `id` from a prior
/v1/messages response (starts with `msg_`)``.
**Expected behavior:** When the persisted session is poisoned and
unrecoverable, the adapter should rotate to a fresh session — the same
fallback path already used for unknown-session errors — and stop
re-persisting the poisoned `session_id`.
**Actual behavior:** The session-rotation fallback only matched the
"unknown session" pattern, so the poisoned session was `--resume`-d
forever. The result event of the failing run still carried `session_id`,
which was being persisted into `agentTaskSessions`, so every subsequent
continuation reloaded the same poisoned id and hit the same 400.
**Reproduction:** Inject any flow that causes Claude Code to emit a
`<synthetic>` placeholder (rate-limit, max-turn exhaustion, transient
upstream failure). The next `--resume` will fail with the 400 and the
agent will not self-recover.
**Scope of fix:** Add a `previous_message_id` 400 detector; route it
through the existing unknown-session fallback; drop the poisoned
`sessionId` and emit `clearSession: true` so the heartbeat service wipes
the persisted row; best-effort delete the local poisoned `.jsonl`.
## What Changed
Two commits:
1. **`adapter-claude-local: auto-rotate session on previous_message_id
400 (synthetic-msg poisoning)`** — detector + execute-time rotation
2. **`adapter-claude-local: guard against persisting poisoned
sessionId`** — validate-before-persist + `clearSession`
Combined diff:
- `parse.ts`: new `isClaudePoisonedPreviousMessageIdError(parsed)`
matching ``/diagnostics\.previous_message_id.*starts with `msg_`/i``
against `parsed.result` and `extractClaudeErrorMessages(parsed)`
- `parse.ts`: `isClaudeTransientUpstreamError()` excludes the new error
from transient classification so it isn't masked as retryable upstream
noise
- `execute.ts`: expand the resume-fallback branch so it triggers on both
`isClaudeUnknownSessionError` and the new
`isClaudePoisonedPreviousMessageIdError`, with a distinct log line
(`"returned a poisoned message-id"` vs `"is unavailable"`)
- `execute.ts`: for local (non-remote) execution targets, best-effort
delete the poisoned `~/.claude/projects/.../{sessionId}.jsonl` before
retrying so the file can't be accidentally resumed by an out-of-band
caller. The `fs.unlink` and follow-up log call are in separate try/catch
blocks so a closed log stream cannot mask a successful unlink (and vice
versa)
- `execute.ts` / `toAdapterResult`: when a result carries the poisoned
400, **drop** `sessionId`/`sessionParams`/`sessionDisplayId` (return
`null`) and emit `clearSession: true` so the heartbeat service's
`resolveNextSessionState` wipes the persisted row. The result also
surfaces `errorCode: "claude_poisoned_previous_message_id"` for
observability
- `docs/adapters/claude-local.md`: runbook entry — symptom,
auto-recovery flow, on-call checklist
- Tests:
- 4 new `parse.test.ts` cases covering positive detection in `result`
and `errors[]`, negative cases, and non-transient classification
- 3 new `claude-local-execute.test.ts` cases: (a) fresh run reports the
poisoned error → sessionId dropped + `clearSession: true`; (b) recovery
retry also reports the poisoned error → same guards apply; (c)
session-rotation success on retry
## Verification
```bash
pnpm --filter @paperclipai/adapter-claude-local exec vitest run src/server/parse.test.ts
pnpm --filter @paperclipai/server exec vitest run src/__tests__/claude-local-execute.test.ts
```
Both suites green locally. This patch is also currently running as a
hot-patch over the published `2026.513.0` adapter on the reporting
deployment — sessions that previously looped indefinitely now
self-recover on the first heartbeat after the 400 surfaces.
## Risks
- Low risk. The detector is conservative (regex over `result` +
`errors[]` only) and the rotation reuses the existing unknown-session
fallback path
- The local-only `fs.unlink` of the poisoned `.jsonl` is wrapped in
`try/catch` and ignored on failure — strictly an optimization; the
server-side session clear is the authoritative reset
- Remote execution targets (`executionTargetIsRemote`) skip the disk
cleanup because the file lives on a remote host that we can't safely
reach from the adapter
- The `clearSession: true` + nulled session fields path is a no-op on
healthy runs; it only fires when the new detector matches, so existing
successful continuations are unaffected
- No DB schema changes, no public API changes, no new dependencies
## Model Used
- Provider: Anthropic Claude
- Model: `claude-opus-4-7` (Opus 4.7)
- Context window: 1M
- Capabilities: extended reasoning, tool use, code execution
- Role: implemented the detector, expanded the fallback branch, added
the persist-guard + `clearSession`, wrote the unit + integration tests,
validated locally, and applied the equivalent hot-patch to the deployed
`2026.513.0` install while this PR is in review
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for similar or duplicate PRs and linked
them — closed #2295, #2361, #3572, #5438 as duplicates of this canonical
fix; complementary fixes #4838 (heartbeat_timer reset) and #4932 (gemini
context-overflow rotation) target different code paths
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots — N/A, adapter-only change
- [x] I have updated relevant documentation
(`docs/adapters/claude-local.md` runbook entry)
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Danial Jawaid <danial.jawaid@gmail.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
This commit is contained in:
@@ -54,8 +54,9 @@ import {
|
||||
isClaudeMaxTurnsResult,
|
||||
isClaudeTransientUpstreamError,
|
||||
isClaudeUnknownSessionError,
|
||||
isClaudePoisonedPreviousMessageIdError,
|
||||
} from "./parse.js";
|
||||
import { prepareClaudeConfigSeed } from "./claude-config.js";
|
||||
import { prepareClaudeConfigSeed, resolveSharedClaudeConfigDir } from "./claude-config.js";
|
||||
import { resolveClaudeDesiredSkillNames } from "./skills.js";
|
||||
import { isBedrockModelId } from "./models.js";
|
||||
import { prepareClaudePromptBundle } from "./prompt-cache.js";
|
||||
@@ -860,9 +861,22 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
|
||||
};
|
||||
})();
|
||||
|
||||
const resolvedSessionId =
|
||||
const rawResolvedSessionId =
|
||||
parsedStream.sessionId ??
|
||||
(asString(parsed.session_id, opts.fallbackSessionId ?? "") || opts.fallbackSessionId);
|
||||
const clearSessionForMaxTurns = isClaudeMaxTurnsResult(parsed);
|
||||
const poisonedPreviousMessageId = isClaudePoisonedPreviousMessageIdError(parsed);
|
||||
const parsedIsError = asBoolean(parsed.is_error, false);
|
||||
const failed = (proc.exitCode ?? 0) !== 0 || parsedIsError;
|
||||
// Validate-before-persist guard: never persist a sessionId whose transcript
|
||||
// is known-poisoned. The Claude CLI keeps an on-disk JSONL keyed by the
|
||||
// session id; if the last entry contains a non-`msg_`-prefixed
|
||||
// `previous_message_id`, every subsequent `--resume` hits a 400 from
|
||||
// /v1/messages and the issue is permanently unrecoverable until the
|
||||
// sessionId is dropped server-side. Drop here so resolveNextSessionState
|
||||
// calls clearTaskSessions on the next heartbeat. See RED-978 / RED-976.
|
||||
const shouldDropSessionForPoison = poisonedPreviousMessageId;
|
||||
const resolvedSessionId = shouldDropSessionForPoison ? null : rawResolvedSessionId;
|
||||
const resolvedSessionParams = resolvedSessionId
|
||||
? ({
|
||||
sessionId: resolvedSessionId,
|
||||
@@ -878,9 +892,6 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
|
||||
...(workspaceRepoRef ? { repoRef: workspaceRepoRef } : {}),
|
||||
} as Record<string, unknown>)
|
||||
: null;
|
||||
const clearSessionForMaxTurns = isClaudeMaxTurnsResult(parsed);
|
||||
const parsedIsError = asBoolean(parsed.is_error, false);
|
||||
const failed = (proc.exitCode ?? 0) !== 0 || parsedIsError;
|
||||
const errorMessage = failed
|
||||
? describeClaudeFailure(parsed) ?? `Claude exited with code ${proc.exitCode ?? -1}`
|
||||
: null;
|
||||
@@ -888,6 +899,7 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
|
||||
failed &&
|
||||
!loginMeta.requiresLogin &&
|
||||
!clearSessionForMaxTurns &&
|
||||
!poisonedPreviousMessageId &&
|
||||
isClaudeTransientUpstreamError({
|
||||
parsed,
|
||||
stdout: proc.stdout,
|
||||
@@ -906,12 +918,15 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
|
||||
? "claude_auth_required"
|
||||
: failed && clearSessionForMaxTurns
|
||||
? "max_turns_exhausted"
|
||||
: failed && poisonedPreviousMessageId
|
||||
? "claude_poisoned_previous_message_id"
|
||||
: transientUpstream
|
||||
? "claude_transient_upstream"
|
||||
: null;
|
||||
const mergedResultJson: Record<string, unknown> = {
|
||||
...parsed,
|
||||
...(failed && clearSessionForMaxTurns ? { stopReason: "max_turns_exhausted" } : {}),
|
||||
...(failed && poisonedPreviousMessageId ? { stopReason: "claude_poisoned_previous_message_id" } : {}),
|
||||
...(transientUpstream ? { errorFamily: "transient_upstream" } : {}),
|
||||
...(transientRetryNotBefore ? { retryNotBefore: transientRetryNotBefore.toISOString() } : {}),
|
||||
...(transientRetryNotBefore ? { transientRetryNotBefore: transientRetryNotBefore.toISOString() } : {}),
|
||||
@@ -937,23 +952,59 @@ export async function execute(ctx: AdapterExecutionContext): Promise<AdapterExec
|
||||
costUsd: parsedStream.costUsd ?? asNumber(parsed.total_cost_usd, 0),
|
||||
resultJson: mergedResultJson,
|
||||
summary: parsedStream.summary || asString(parsed.result, ""),
|
||||
clearSession: clearSessionForMaxTurns || Boolean(opts.clearSessionOnMissingSession && !resolvedSessionId),
|
||||
clearSession:
|
||||
clearSessionForMaxTurns ||
|
||||
// Clear-on-error: a poisoned previous_message_id is a deterministic
|
||||
// state error. Force the server to drop persisted session state for
|
||||
// this issue so the next continuation starts from a clean slate.
|
||||
poisonedPreviousMessageId ||
|
||||
Boolean(opts.clearSessionOnMissingSession && !resolvedSessionId),
|
||||
};
|
||||
};
|
||||
|
||||
try {
|
||||
const initial = await runAttempt(sessionId ?? null);
|
||||
if (
|
||||
const sessionErrorKind =
|
||||
sessionId &&
|
||||
!initial.proc.timedOut &&
|
||||
(initial.proc.exitCode ?? 0) !== 0 &&
|
||||
initial.parsed &&
|
||||
isClaudeUnknownSessionError(initial.parsed)
|
||||
) {
|
||||
initial.parsed
|
||||
? isClaudeUnknownSessionError(initial.parsed)
|
||||
? "unknown"
|
||||
: isClaudePoisonedPreviousMessageIdError(initial.parsed)
|
||||
? "poisoned"
|
||||
: null
|
||||
: null;
|
||||
|
||||
if (sessionErrorKind !== null) {
|
||||
const reason =
|
||||
sessionErrorKind === "poisoned"
|
||||
? "returned a poisoned message-id"
|
||||
: "is unavailable";
|
||||
await onLog(
|
||||
"stdout",
|
||||
`[paperclip] Claude resume session "${sessionId}" is unavailable; retrying with a fresh session.\n`,
|
||||
`[paperclip] Claude resume session "${sessionId}" ${reason}; retrying with a fresh session.\n`,
|
||||
);
|
||||
if (sessionErrorKind === "poisoned" && !executionTargetIsRemote) {
|
||||
const claudeConfigDir = resolveSharedClaudeConfigDir(effectiveEnv);
|
||||
// Mirrors Claude Code's project-dir encoding: non-alphanumeric chars become "-"; existing hyphens pass through.
|
||||
const encodedCwd = effectiveExecutionCwd.replace(/[^a-zA-Z0-9-]/g, "-");
|
||||
const poisonedJsonlPath = path.join(claudeConfigDir, "projects", encodedCwd, `${sessionId}.jsonl`);
|
||||
let unlinked = false;
|
||||
try {
|
||||
await fs.unlink(poisonedJsonlPath);
|
||||
unlinked = true;
|
||||
} catch {
|
||||
// best-effort; session is cleared server-side regardless
|
||||
}
|
||||
if (unlinked) {
|
||||
try {
|
||||
await onLog("stdout", `[paperclip] Removed poisoned session file: ${poisonedJsonlPath}\n`);
|
||||
} catch {
|
||||
// log stream may be closed; the unlink already succeeded
|
||||
}
|
||||
}
|
||||
}
|
||||
const retry = await runAttempt(null);
|
||||
return toAdapterResult(retry, { fallbackSessionId: null, clearSessionOnMissingSession: true });
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user