fix(server): enforce agent secret binding sync across lifecycle flows (#8307)
## Thinking Path > - Paperclip is the control plane people use to create, configure, and run AI agents for work. > - This change sits in the server-side agent lifecycle and secret-binding subsystem, where adapter config `env` entries can reference company secrets. > - An incident (while trying to configure a Novita sandbox) showed that an agent can reach a broken runtime state if `adapterConfig.env` contains `secret_ref` entries but the matching `company_secret_bindings` rows are missing. > - The immediate run-path guard and error-surfacing work made the failure diagnosable, but they did not fully prevent new broken agents from being created. > - The risk came from create and approval flows being responsible for remembering to sync bindings at each call site, which is easy to miss as new flows are added. > - This pull request moves the invariant into `agentService` create/update/activate paths, keeps the existing hire-flow fix, and adds regression coverage for create, update, and legacy pending-approval recovery. > - The benefit is that agent secret binding integrity is enforced closer to the data mutation point, so future callers inherit the protection automatically. ## Linked Issues or Issue Description Refs #8309 ### What happened? A Paperclip agent could persist `adapterConfig.env` `secret_ref` entries without matching agent-scoped `company_secret_bindings` rows. When that happened, the config UI could still look configured, but the real run path failed pre-dispatch because the secret was not actually bound to that agent. ### Expected behavior Every normal agent create, config-update, and pending-approval activation flow should leave the agent with secret bindings that match its persisted secret-ref env config. ### Steps to reproduce 1. Create or activate an agent through a flow that persists `adapterConfig.env` secret refs without synchronizing `company_secret_bindings`. 2. Observe that the config state can still appear populated. 3. Start a run for that agent. 4. Observe that pre-dispatch binding validation fails because the secret reference exists but the agent binding does not. ### Deployment mode Local dev (`pnpm dev`) ### Installation method Built from source (`pnpm dev` / `pnpm build`) ### Agent adapter(s) involved - Claude Code - Not adapter-specific (core bug) ### Database mode Embedded PGlite / embedded local dev database flow ### Access context Board (human operator) created or approved the agent; agent runtime later consumed the config. ### Additional context This PR focuses on preventing new broken states from normal service flows and on backfilling the covered legacy pending-approval activation path. ## What Changed - Kept the existing branch-local hire-flow fix that synchronized bindings for route and approval paths. - Moved the binding integrity invariant into `agentService.create()`, `agentService.update()` when `adapterConfig` changes, and `agentService.activatePendingApproval()`. - Added `server/src/__tests__/agents-service-secret-bindings.test.ts` covering create-time sync, update-time resync, and backfill for legacy pending-approval agents. - Removed now-redundant route-layer and approval-layer binding sync calls once the service layer became authoritative. - Simplified the affected unit tests so route/approval tests no longer assert service-owned binding writes directly. ## Verification - `pnpm --filter @paperclipai/server typecheck` - `pnpm exec vitest run server/src/__tests__/agents-service-secret-bindings.test.ts server/src/__tests__/approvals-service.test.ts server/src/__tests__/agent-skills-routes.test.ts` ## Risks - Low to medium risk. - This changes where secret-binding synchronization is enforced, so any unexpected caller that relied on upper-layer manual sync behavior could behave differently. - Agent create/update/activation flows now perform binding synchronization consistently, which adds binding-table writes at those mutation points. - This PR does not retroactively scan and heal every already-broken historical agent row; it prevents and backfills through the covered service flows. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - OpenAI Codex / GPT-5 Codex class model via `codex_local` - Session model family: GPT-5 Codex - Tool-assisted coding with shell, git, HTTP, and local test execution - Reasoning mode: medium interactive tool-use workflow ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
@@ -0,0 +1,26 @@
|
||||
interface AgentSecretBindingSyncService {
|
||||
syncEnvBindingsForTarget?: (
|
||||
companyId: string,
|
||||
target: { targetType: "agent"; targetId: string; pathPrefix?: string },
|
||||
envValue: unknown,
|
||||
) => Promise<unknown>;
|
||||
}
|
||||
|
||||
function asRecord(value: unknown): Record<string, unknown> | null {
|
||||
if (typeof value !== "object" || value === null || Array.isArray(value)) return null;
|
||||
return value as Record<string, unknown>;
|
||||
}
|
||||
|
||||
export async function syncAgentAdapterEnvBindings(input: {
|
||||
secretsSvc: AgentSecretBindingSyncService;
|
||||
companyId: string;
|
||||
agentId: string;
|
||||
adapterConfig: unknown;
|
||||
}) {
|
||||
const envValue = asRecord(asRecord(input.adapterConfig)?.env);
|
||||
await input.secretsSvc.syncEnvBindingsForTarget?.(
|
||||
input.companyId,
|
||||
{ targetType: "agent", targetId: input.agentId },
|
||||
envValue,
|
||||
);
|
||||
}
|
||||
@@ -24,8 +24,10 @@ import {
|
||||
type AgentEligibilityAgent,
|
||||
} from "@paperclipai/shared";
|
||||
import { conflict, notFound, unprocessable } from "../errors.js";
|
||||
import { syncAgentAdapterEnvBindings } from "./agent-secret-bindings.js";
|
||||
import { normalizeAgentPermissions } from "./agent-permissions.js";
|
||||
import { REDACTED_EVENT_VALUE, sanitizeRecord } from "../redaction.js";
|
||||
import { secretService } from "./secrets.js";
|
||||
|
||||
function hashToken(token: string) {
|
||||
return createHash("sha256").update(token).digest("hex");
|
||||
@@ -218,6 +220,8 @@ export function deduplicateAgentName(
|
||||
}
|
||||
|
||||
export function agentService(db: Db) {
|
||||
const secretsSvc = secretService(db);
|
||||
|
||||
function currentUtcMonthWindow(now = new Date()) {
|
||||
const year = now.getUTCFullYear();
|
||||
const month = now.getUTCMonth();
|
||||
@@ -371,6 +375,19 @@ export function agentService(db: Db) {
|
||||
}
|
||||
}
|
||||
|
||||
async function syncAgentSecretBindings(
|
||||
agent: { id: string; companyId: string; adapterConfig: unknown },
|
||||
dbClient: Db = db,
|
||||
) {
|
||||
const scopedSecretsSvc = dbClient === db ? secretsSvc : secretService(dbClient);
|
||||
await syncAgentAdapterEnvBindings({
|
||||
secretsSvc: scopedSecretsSvc,
|
||||
companyId: agent.companyId,
|
||||
agentId: agent.id,
|
||||
adapterConfig: agent.adapterConfig,
|
||||
});
|
||||
}
|
||||
|
||||
async function updateAgent(
|
||||
id: string,
|
||||
data: Partial<typeof agents.$inferInsert>,
|
||||
@@ -415,33 +432,45 @@ export function agentService(db: Db) {
|
||||
const shouldRecordRevision = Boolean(options?.recordRevision) && hasConfigPatchFields(normalizedPatch);
|
||||
const beforeConfig = shouldRecordRevision ? buildConfigSnapshot(existing) : null;
|
||||
|
||||
const updated = await db
|
||||
.update(agents)
|
||||
.set({ ...normalizedPatch, updatedAt: new Date() })
|
||||
.where(eq(agents.id, id))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
const normalizedUpdated = updated ? await getById(updated.id) : null;
|
||||
return db.transaction(async (tx) => {
|
||||
const txDb = tx as unknown as Db;
|
||||
const updated = await tx
|
||||
.update(agents)
|
||||
.set({ ...normalizedPatch, updatedAt: new Date() })
|
||||
.where(eq(agents.id, id))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!updated) return null;
|
||||
|
||||
if (normalizedUpdated && shouldRecordRevision && beforeConfig) {
|
||||
const afterConfig = buildConfigSnapshot(normalizedUpdated);
|
||||
const changedKeys = diffConfigSnapshot(beforeConfig, afterConfig);
|
||||
if (changedKeys.length > 0) {
|
||||
await db.insert(agentConfigRevisions).values({
|
||||
companyId: normalizedUpdated.companyId,
|
||||
agentId: normalizedUpdated.id,
|
||||
createdByAgentId: options?.recordRevision?.createdByAgentId ?? null,
|
||||
createdByUserId: options?.recordRevision?.createdByUserId ?? null,
|
||||
source: options?.recordRevision?.source ?? "patch",
|
||||
rolledBackFromRevisionId: options?.recordRevision?.rolledBackFromRevisionId ?? null,
|
||||
changedKeys,
|
||||
beforeConfig: beforeConfig as unknown as Record<string, unknown>,
|
||||
afterConfig: afterConfig as unknown as Record<string, unknown>,
|
||||
});
|
||||
if (Object.prototype.hasOwnProperty.call(normalizedPatch, "adapterConfig")) {
|
||||
await syncAgentSecretBindings(updated, txDb);
|
||||
}
|
||||
}
|
||||
|
||||
return normalizedUpdated;
|
||||
const normalizedUpdated = await agentService(txDb).getById(updated.id);
|
||||
if (!normalizedUpdated) {
|
||||
throw notFound("Agent not found");
|
||||
}
|
||||
|
||||
if (shouldRecordRevision && beforeConfig) {
|
||||
const afterConfig = buildConfigSnapshot(normalizedUpdated);
|
||||
const changedKeys = diffConfigSnapshot(beforeConfig, afterConfig);
|
||||
if (changedKeys.length > 0) {
|
||||
await tx.insert(agentConfigRevisions).values({
|
||||
companyId: normalizedUpdated.companyId,
|
||||
agentId: normalizedUpdated.id,
|
||||
createdByAgentId: options?.recordRevision?.createdByAgentId ?? null,
|
||||
createdByUserId: options?.recordRevision?.createdByUserId ?? null,
|
||||
source: options?.recordRevision?.source ?? "patch",
|
||||
rolledBackFromRevisionId: options?.recordRevision?.rolledBackFromRevisionId ?? null,
|
||||
changedKeys,
|
||||
beforeConfig: beforeConfig as unknown as Record<string, unknown>,
|
||||
afterConfig: afterConfig as unknown as Record<string, unknown>,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return normalizedUpdated;
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
@@ -474,13 +503,20 @@ export function agentService(db: Db) {
|
||||
const role = data.role ?? "general";
|
||||
const normalizedPermissions = normalizeAgentPermissions(data.permissions, role);
|
||||
const runtimeConfig = normalizeRuntimeConfigForNewAgent(data.runtimeConfig);
|
||||
const created = await db
|
||||
.insert(agents)
|
||||
.values({ ...data, name: uniqueName, companyId, role, permissions: normalizedPermissions, runtimeConfig })
|
||||
.returning()
|
||||
.then((rows) => rows[0]);
|
||||
|
||||
return requireGetById(created.id);
|
||||
return db.transaction(async (tx) => {
|
||||
const txDb = tx as unknown as Db;
|
||||
const created = await tx
|
||||
.insert(agents)
|
||||
.values({ ...data, name: uniqueName, companyId, role, permissions: normalizedPermissions, runtimeConfig })
|
||||
.returning()
|
||||
.then((rows) => rows[0]);
|
||||
await syncAgentSecretBindings(created, txDb);
|
||||
const normalizedCreated = await agentService(txDb).getById(created.id);
|
||||
if (!normalizedCreated) {
|
||||
throw notFound("Agent not found");
|
||||
}
|
||||
return normalizedCreated;
|
||||
});
|
||||
},
|
||||
|
||||
update: updateAgent,
|
||||
@@ -496,6 +532,7 @@ export function agentService(db: Db) {
|
||||
status: "paused",
|
||||
pauseReason: reason,
|
||||
pausedAt: new Date(),
|
||||
errorReason: null,
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(eq(agents.id, id))
|
||||
@@ -518,6 +555,7 @@ export function agentService(db: Db) {
|
||||
status: "idle",
|
||||
pauseReason: null,
|
||||
pausedAt: null,
|
||||
errorReason: null,
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(eq(agents.id, id))
|
||||
@@ -543,6 +581,7 @@ export function agentService(db: Db) {
|
||||
status: "idle",
|
||||
pauseReason: null,
|
||||
pausedAt: null,
|
||||
errorReason: null,
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(and(eq(agents.id, id), eq(agents.status, "error")))
|
||||
@@ -565,6 +604,7 @@ export function agentService(db: Db) {
|
||||
status: "terminated",
|
||||
pauseReason: null,
|
||||
pausedAt: null,
|
||||
errorReason: null,
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
.where(eq(agents.id, id));
|
||||
@@ -611,15 +651,25 @@ export function agentService(db: Db) {
|
||||
},
|
||||
|
||||
activatePendingApproval: async (id: string) => {
|
||||
const updated = await db
|
||||
.update(agents)
|
||||
.set({ status: "idle", updatedAt: new Date() })
|
||||
.where(and(eq(agents.id, id), eq(agents.status, "pending_approval")))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
const activatedAgent = await db.transaction(async (tx) => {
|
||||
const txDb = tx as unknown as Db;
|
||||
const updated = await tx
|
||||
.update(agents)
|
||||
.set({ status: "idle", updatedAt: new Date() })
|
||||
.where(and(eq(agents.id, id), eq(agents.status, "pending_approval")))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!updated) return null;
|
||||
await syncAgentSecretBindings(updated, txDb);
|
||||
const agent = await agentService(txDb).getById(updated.id);
|
||||
if (!agent) {
|
||||
throw notFound("Agent not found");
|
||||
}
|
||||
return agent;
|
||||
});
|
||||
|
||||
if (updated) {
|
||||
return { agent: await requireGetById(updated.id), activated: true };
|
||||
if (activatedAgent) {
|
||||
return { agent: activatedAgent, activated: true };
|
||||
}
|
||||
|
||||
const existing = await getById(id);
|
||||
|
||||
@@ -71,7 +71,7 @@ import { trackAgentFirstHeartbeat } from "@paperclipai/shared/telemetry";
|
||||
import { getTelemetryClient } from "../telemetry.js";
|
||||
import { companySkillService } from "./company-skills.js";
|
||||
import { budgetService, type BudgetEnforcementScope } from "./budgets.js";
|
||||
import { secretService } from "./secrets.js";
|
||||
import { secretService, type MissingRuntimeBinding } from "./secrets.js";
|
||||
import { resolveDefaultAgentWorkspaceDir, resolveManagedProjectWorkspaceDir } from "../home-paths.js";
|
||||
import {
|
||||
buildHeartbeatRunIssueComment,
|
||||
@@ -254,6 +254,8 @@ const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_WAKE_REASON = "transient_failure_retry";
|
||||
const BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS = BOUNDED_TRANSIENT_HEARTBEAT_RETRY_DELAYS_MS.length;
|
||||
const WORKSPACE_VALIDATION_FAILURE_CODE = "workspace_validation_failed";
|
||||
const WORKSPACE_VALIDATION_RECOVERY_CAUSE = "workspace_validation_failed";
|
||||
const CONFIGURATION_INCOMPLETE_FAILURE_CODE = "configuration_incomplete";
|
||||
const CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE = "configuration_incomplete";
|
||||
// Keep this in sync with local adapters that require a git workspace before launch.
|
||||
const GIT_SENSITIVE_LOCAL_ADAPTER_TYPES = new Set([
|
||||
"acpx_local",
|
||||
@@ -296,6 +298,20 @@ export class WorkspaceValidationFailure extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
// Pre-dispatch gate outcome: required secret/env bindings are missing, so the
|
||||
// run must not be dispatched. Surfaced as a configuration-incomplete blocker
|
||||
// routed to a human owner instead of N opaque dispatched-then-failed runs.
|
||||
export class ConfigurationIncompleteFailure extends Error {
|
||||
code = CONFIGURATION_INCOMPLETE_FAILURE_CODE;
|
||||
resultJson: Record<string, unknown>;
|
||||
|
||||
constructor(message: string, resultJson: Record<string, unknown>) {
|
||||
super(message);
|
||||
this.name = "ConfigurationIncompleteFailure";
|
||||
this.resultJson = resultJson;
|
||||
}
|
||||
}
|
||||
|
||||
function resolveCodexTransientFallbackMode(attempt: number): CodexTransientFallbackMode {
|
||||
if (attempt <= 1) return "same_session";
|
||||
if (attempt === 2) return "safer_invocation";
|
||||
@@ -381,9 +397,16 @@ const INLINE_BASE64_IMAGE_DATA_RE = /("type":"image","source":\{"type":"base64",
|
||||
|
||||
type RuntimeConfigSecretResolver = Pick<
|
||||
ReturnType<typeof secretService>,
|
||||
"resolveAdapterConfigForRuntime" | "resolveEnvBindings"
|
||||
"resolveAdapterConfigForRuntime" | "resolveEnvBindings" | "collectMissingRuntimeBindings"
|
||||
>;
|
||||
|
||||
function formatMissingBindingForOperator(missing: MissingRuntimeBinding): string {
|
||||
const secretLabel = missing.secretName
|
||||
? `"${missing.secretName}"`
|
||||
: missing.secretId;
|
||||
return `secret ${secretLabel} not bound at ${missing.consumerType} ${missing.configPath}`;
|
||||
}
|
||||
|
||||
const LOW_TRUST_SENSITIVE_ENV_KEY_RE =
|
||||
/(api[-_]?key|access[-_]?token|auth(?:_?token)?|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring)/i;
|
||||
|
||||
@@ -449,6 +472,54 @@ export async function resolveExecutionRunAdapterConfig(input: {
|
||||
assertLowTrustEnvConfigAllowed(projectEnv, "project.env");
|
||||
assertLowTrustEnvConfigAllowed(routineEnv, "routine.env");
|
||||
}
|
||||
// Pre-dispatch binding-validation gate: detect declared secret refs that have
|
||||
// no binding before resolving any secret value. Missing bindings short-circuit
|
||||
// to a configuration-incomplete blocker routed to a human owner instead of a
|
||||
// dispatched-then-failed run (which previously surfaced as opaque setup_failed).
|
||||
if (typeof input.secretsSvc.collectMissingRuntimeBindings === "function") {
|
||||
const missingBindings: MissingRuntimeBinding[] = [];
|
||||
if (input.agentId) {
|
||||
missingBindings.push(
|
||||
...(await input.secretsSvc.collectMissingRuntimeBindings(
|
||||
input.companyId,
|
||||
parseObject(executionRunConfig.env),
|
||||
{ consumerType: "agent", consumerId: input.agentId },
|
||||
)),
|
||||
);
|
||||
}
|
||||
if (projectEnv && input.projectId) {
|
||||
missingBindings.push(
|
||||
...(await input.secretsSvc.collectMissingRuntimeBindings(
|
||||
input.companyId,
|
||||
projectEnv,
|
||||
{ consumerType: "project", consumerId: input.projectId },
|
||||
)),
|
||||
);
|
||||
}
|
||||
if (routineEnv && input.routineId) {
|
||||
missingBindings.push(
|
||||
...(await input.secretsSvc.collectMissingRuntimeBindings(
|
||||
input.companyId,
|
||||
routineEnv,
|
||||
{ consumerType: "routine", consumerId: input.routineId },
|
||||
)),
|
||||
);
|
||||
}
|
||||
if (missingBindings.length > 0) {
|
||||
const detail = missingBindings.map(formatMissingBindingForOperator).join("; ");
|
||||
throw new ConfigurationIncompleteFailure(`configuration incomplete: ${detail}`, {
|
||||
configurationIncomplete: {
|
||||
reason: "secret_binding_missing",
|
||||
companyId: input.companyId,
|
||||
agentId: input.agentId ?? null,
|
||||
issueId: input.issueId ?? null,
|
||||
projectId: input.projectId ?? null,
|
||||
routineId: input.routineId ?? null,
|
||||
missingBindings,
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
const { config: resolvedConfig, secretKeys, manifest } = await input.secretsSvc.resolveAdapterConfigForRuntime(
|
||||
input.companyId,
|
||||
executionRunConfig,
|
||||
@@ -939,6 +1010,16 @@ function isWorkspaceValidationFailedRun(
|
||||
return run?.errorCode === WORKSPACE_VALIDATION_FAILURE_CODE;
|
||||
}
|
||||
|
||||
function isConfigurationIncompleteFailure(error: unknown): error is ConfigurationIncompleteFailure {
|
||||
return error instanceof ConfigurationIncompleteFailure;
|
||||
}
|
||||
|
||||
function isConfigurationIncompleteFailedRun(
|
||||
run: Pick<typeof heartbeatRuns.$inferSelect, "errorCode"> | null | undefined,
|
||||
) {
|
||||
return run?.errorCode === CONFIGURATION_INCOMPLETE_FAILURE_CODE;
|
||||
}
|
||||
|
||||
async function hasGitMetadata(cwd: string | null | undefined) {
|
||||
const normalized = readNonEmptyString(cwd);
|
||||
if (!normalized) return false;
|
||||
@@ -7199,9 +7280,17 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
return cancelled;
|
||||
}
|
||||
|
||||
function truncateAgentErrorReason(reason: string | null | undefined): string | null {
|
||||
if (!reason) return null;
|
||||
const trimmed = reason.trim();
|
||||
if (!trimmed) return null;
|
||||
return trimmed.length > 500 ? `${trimmed.slice(0, 499)}…` : trimmed;
|
||||
}
|
||||
|
||||
async function finalizeAgentStatus(
|
||||
agentId: string,
|
||||
outcome: "succeeded" | "failed" | "cancelled" | "timed_out",
|
||||
failureReason?: string | null,
|
||||
) {
|
||||
const existing = await getAgent(agentId);
|
||||
if (!existing) return;
|
||||
@@ -7224,6 +7313,10 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
.update(agents)
|
||||
.set({
|
||||
status: nextStatus,
|
||||
// Persist a human-readable reason on the agent record when it enters
|
||||
// error so operators see it on the agent page without digging into run
|
||||
// events; clear it whenever the agent leaves error.
|
||||
errorReason: nextStatus === "error" ? truncateAgentErrorReason(failureReason) : null,
|
||||
lastHeartbeatAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
})
|
||||
@@ -7586,7 +7679,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
},
|
||||
});
|
||||
|
||||
await finalizeAgentStatus(run.agentId, "failed");
|
||||
await finalizeAgentStatus(run.agentId, "failed", baseMessage);
|
||||
await startNextQueuedRunForAgent(run.agentId);
|
||||
runningProcesses.delete(run.id);
|
||||
reaped.push(run.id);
|
||||
@@ -9439,14 +9532,20 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
}
|
||||
}
|
||||
}
|
||||
await finalizeAgentStatus(agent.id, outcome);
|
||||
await finalizeAgentStatus(
|
||||
agent.id,
|
||||
outcome,
|
||||
outcome === "succeeded" ? null : (adapterResult.errorMessage ?? null),
|
||||
);
|
||||
} catch (err) {
|
||||
const message = redactCurrentUserText(
|
||||
err instanceof Error ? err.message : "Unknown adapter failure",
|
||||
await getCurrentUserRedactionOptions(),
|
||||
);
|
||||
const workspaceValidationFailure = isWorkspaceValidationFailure(err) ? err : null;
|
||||
const failureErrorCode = workspaceValidationFailure?.code ?? "adapter_failed";
|
||||
const configurationIncompleteFailure = isConfigurationIncompleteFailure(err) ? err : null;
|
||||
const failureErrorCode =
|
||||
workspaceValidationFailure?.code ?? configurationIncompleteFailure?.code ?? "adapter_failed";
|
||||
logger.error({ err, runId }, "heartbeat execution failed");
|
||||
|
||||
let logSummary: { bytes: number; sha256?: string; compressed: boolean } | null = null;
|
||||
@@ -9472,7 +9571,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
resultJson: mergeRunStopMetadataForAgent(agent, "failed", {
|
||||
errorCode: failureErrorCode,
|
||||
errorMessage: message,
|
||||
resultJson: workspaceValidationFailure?.resultJson ?? null,
|
||||
resultJson: workspaceValidationFailure?.resultJson ?? configurationIncompleteFailure?.resultJson ?? null,
|
||||
}),
|
||||
stdoutExcerpt,
|
||||
stderrExcerpt,
|
||||
@@ -9507,7 +9606,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
});
|
||||
const livenessRun = await classifyAndPersistRunLiveness(failedRun) ?? failedRun;
|
||||
await refreshContinuationSummaryForRun(livenessRun, agent);
|
||||
if (!isWorkspaceValidationFailedRun(livenessRun)) {
|
||||
if (!isWorkspaceValidationFailedRun(livenessRun) && !isConfigurationIncompleteFailedRun(livenessRun)) {
|
||||
await finalizeIssueCommentPolicy(livenessRun, agent);
|
||||
}
|
||||
await releaseIssueExecutionAndPromote(livenessRun);
|
||||
@@ -9535,22 +9634,31 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
}
|
||||
}
|
||||
|
||||
await finalizeAgentStatus(agent.id, "failed");
|
||||
await finalizeAgentStatus(agent.id, "failed", message);
|
||||
}
|
||||
} catch (outerErr) {
|
||||
// Setup code before adapter.execute threw (e.g. ensureRuntimeState, resolveWorkspaceForRun).
|
||||
// The inner catch did not fire, so we must record the failure here.
|
||||
const message = outerErr instanceof Error ? outerErr.message : "Unknown setup failure";
|
||||
const message = redactCurrentUserText(
|
||||
outerErr instanceof Error ? outerErr.message : "Unknown setup failure",
|
||||
await getCurrentUserRedactionOptions(),
|
||||
);
|
||||
// A missing secret/env binding is a known pre-dispatch configuration gap,
|
||||
// not an opaque setup crash. Surface it with its own errorCode so the
|
||||
// recovery path routes it to a human owner instead of looping retries.
|
||||
const configurationIncompleteSetupFailure = isConfigurationIncompleteFailure(outerErr) ? outerErr : null;
|
||||
const setupFailureErrorCode = configurationIncompleteSetupFailure?.code ?? "setup_failed";
|
||||
logger.error({ err: outerErr, runId }, "heartbeat execution setup failed");
|
||||
const setupFailureAgent = await getAgent(run.agentId).catch(() => null);
|
||||
const setupFailureWrite = await setRunStatusIfRunning(runId, "failed", {
|
||||
error: message,
|
||||
errorCode: "setup_failed",
|
||||
errorCode: setupFailureErrorCode,
|
||||
finishedAt: new Date(),
|
||||
...(setupFailureAgent ? {
|
||||
resultJson: mergeRunStopMetadataForAgent(setupFailureAgent, "failed", {
|
||||
errorCode: "setup_failed",
|
||||
errorCode: setupFailureErrorCode,
|
||||
errorMessage: message,
|
||||
resultJson: configurationIncompleteSetupFailure?.resultJson ?? null,
|
||||
}),
|
||||
} : {}),
|
||||
}).catch(() => ({ run: null, updated: false as const }));
|
||||
@@ -9583,7 +9691,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
const failedAgent = setupFailureAgent ?? await getAgent(run.agentId).catch(() => null);
|
||||
if (failedAgent) {
|
||||
await refreshContinuationSummaryForRun(livenessRun, failedAgent).catch(() => undefined);
|
||||
if (!isWorkspaceValidationFailedRun(livenessRun)) {
|
||||
if (!isWorkspaceValidationFailedRun(livenessRun) && !isConfigurationIncompleteFailedRun(livenessRun)) {
|
||||
await finalizeIssueCommentPolicy(livenessRun, failedAgent).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
@@ -9593,7 +9701,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
// path owned the terminal transition. If another path already finalized
|
||||
// the run, keep that terminal outcome authoritative.
|
||||
if (setupFailureWrite.updated) {
|
||||
await finalizeAgentStatus(run.agentId, "failed").catch(() => undefined);
|
||||
await finalizeAgentStatus(run.agentId, "failed", message).catch(() => undefined);
|
||||
}
|
||||
} finally {
|
||||
const latestRun = await getRun(run.id).catch(() => null);
|
||||
@@ -9641,6 +9749,17 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
);
|
||||
}
|
||||
|
||||
function buildConfigurationIncompleteRecoveryComment(input: {
|
||||
latestRun: Pick<typeof heartbeatRuns.$inferSelect, "error" | "errorCode"> | null | undefined;
|
||||
}) {
|
||||
const failureSummary = summarizeRunFailureForIssueComment(input.latestRun);
|
||||
return (
|
||||
"Paperclip stopped before dispatching the adapter because required secret/env bindings are missing. " +
|
||||
`Resolving them as a runtime failure would only produce repeated opaque setup failures.${failureSummary ?? ""} ` +
|
||||
"Moving it to `blocked` with a source-scoped recovery action so an operator can bind the missing secret(s) before resuming."
|
||||
);
|
||||
}
|
||||
|
||||
async function releaseIssueExecutionAndPromote(run: typeof heartbeatRuns.$inferSelect) {
|
||||
const runContext = parseObject(run.contextSnapshot);
|
||||
const contextIssueId = readNonEmptyString(runContext.issueId);
|
||||
@@ -9761,17 +9880,22 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
// Sibling lock cleanup is already done above; only the primary issue carries
|
||||
// the recovery surface because the comment is attached to a single issue.
|
||||
if (
|
||||
isWorkspaceValidationFailedRun(run) &&
|
||||
(isWorkspaceValidationFailedRun(run) || isConfigurationIncompleteFailedRun(run)) &&
|
||||
(issue.status === "todo" || issue.status === "in_progress") &&
|
||||
!issue.assigneeUserId &&
|
||||
issue.assigneeAgentId === run.agentId
|
||||
) {
|
||||
const configurationIncomplete = isConfigurationIncompleteFailedRun(run);
|
||||
return {
|
||||
kind: "blocked" as const,
|
||||
issue,
|
||||
previousStatus: issue.status,
|
||||
comment: buildWorkspaceValidationRecoveryComment({ latestRun: run }),
|
||||
recoveryCause: WORKSPACE_VALIDATION_RECOVERY_CAUSE,
|
||||
comment: configurationIncomplete
|
||||
? buildConfigurationIncompleteRecoveryComment({ latestRun: run })
|
||||
: buildWorkspaceValidationRecoveryComment({ latestRun: run }),
|
||||
recoveryCause: configurationIncomplete
|
||||
? CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE
|
||||
: WORKSPACE_VALIDATION_RECOVERY_CAUSE,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -10057,21 +10181,29 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
!recoveryAgentInvokable ||
|
||||
!recoveryAgent ||
|
||||
isWorkspaceValidationFailedRun(run) ||
|
||||
isConfigurationIncompleteFailedRun(run) ||
|
||||
didAutomaticRecoveryFail(run, issue.status === "todo" ? "assignment_recovery" : "issue_continuation_needed");
|
||||
if (shouldBlockImmediately) {
|
||||
const workspaceValidationFailure = isWorkspaceValidationFailedRun(run);
|
||||
const configurationIncompleteFailure = isConfigurationIncompleteFailedRun(run);
|
||||
const comment = workspaceValidationFailure
|
||||
? buildWorkspaceValidationRecoveryComment({ latestRun: run })
|
||||
: buildImmediateExecutionPathRecoveryComment({
|
||||
status: issue.status as "todo" | "in_progress",
|
||||
latestRun: run,
|
||||
});
|
||||
: configurationIncompleteFailure
|
||||
? buildConfigurationIncompleteRecoveryComment({ latestRun: run })
|
||||
: buildImmediateExecutionPathRecoveryComment({
|
||||
status: issue.status as "todo" | "in_progress",
|
||||
latestRun: run,
|
||||
});
|
||||
return {
|
||||
kind: "blocked" as const,
|
||||
issue,
|
||||
previousStatus: issue.status,
|
||||
comment,
|
||||
recoveryCause: workspaceValidationFailure ? WORKSPACE_VALIDATION_RECOVERY_CAUSE : undefined,
|
||||
recoveryCause: workspaceValidationFailure
|
||||
? WORKSPACE_VALIDATION_RECOVERY_CAUSE
|
||||
: configurationIncompleteFailure
|
||||
? CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE
|
||||
: undefined,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -10157,7 +10289,9 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
recoveryCause:
|
||||
promotionResult.recoveryCause === WORKSPACE_VALIDATION_RECOVERY_CAUSE
|
||||
? WORKSPACE_VALIDATION_RECOVERY_CAUSE
|
||||
: undefined,
|
||||
: promotionResult.recoveryCause === CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE
|
||||
? CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE
|
||||
: undefined,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -119,6 +119,7 @@ type SuccessfulLatestIssueRun = NonNullable<LatestIssueRun> & { status: "succeed
|
||||
type StrandedRecoveryCause =
|
||||
| "stranded_assigned_issue"
|
||||
| "workspace_validation_failed"
|
||||
| "configuration_incomplete"
|
||||
| typeof SUCCESSFUL_RUN_MISSING_STATE_REASON;
|
||||
|
||||
type SuccessfulRunHandoffRecoveryEvidence = {
|
||||
@@ -2231,6 +2232,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
||||
? "missing_disposition" as const
|
||||
: cause === "workspace_validation_failed"
|
||||
? "workspace_validation" as const
|
||||
: cause === "configuration_incomplete"
|
||||
? "configuration_validation" as const
|
||||
: "stranded_assigned_issue" as const;
|
||||
}
|
||||
|
||||
@@ -2306,11 +2309,13 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
||||
? "Choose and record a valid issue disposition without copying transcript content."
|
||||
: recoveryCause === "workspace_validation_failed"
|
||||
? "Repair the source issue workspace link, project workspace cwd, or git checkout before resuming adapter execution."
|
||||
: recoveryCause === "configuration_incomplete"
|
||||
? "Bind the missing secret(s) named in the run failure to the agent/project/routine env before resuming adapter execution."
|
||||
: "Restore a live execution path, fix the runtime/adapter failure, or record an intentional manual resolution.",
|
||||
wakePolicy: recoveryCause === "workspace_validation_failed"
|
||||
wakePolicy: recoveryCause === "workspace_validation_failed" || recoveryCause === "configuration_incomplete"
|
||||
? {
|
||||
type: "manual_repair_required",
|
||||
reason: "workspace_validation_failed",
|
||||
reason: recoveryCause,
|
||||
ownerAgentId,
|
||||
}
|
||||
: ownerAgentId
|
||||
@@ -2338,6 +2343,7 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
||||
recoveryCause: StrandedRecoveryCause;
|
||||
}) {
|
||||
if (input.recoveryCause === "workspace_validation_failed") return;
|
||||
if (input.recoveryCause === "configuration_incomplete") return;
|
||||
if (!input.action.ownerAgentId) return;
|
||||
await deps.enqueueWakeup(input.action.ownerAgentId, {
|
||||
source: "assignment",
|
||||
@@ -2543,7 +2549,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
||||
|
||||
const shouldPostEscalationComment =
|
||||
recoveryAction.attemptCount === 1 ||
|
||||
input.recoveryCause === "workspace_validation_failed";
|
||||
input.recoveryCause === "workspace_validation_failed" ||
|
||||
input.recoveryCause === "configuration_incomplete";
|
||||
if (shouldPostEscalationComment) {
|
||||
const escalationCommentMarker = `Recovery action: \`${recoveryAction.id}\``;
|
||||
|
||||
@@ -2597,6 +2604,8 @@ export function recoveryService(db: Db, deps: { enqueueWakeup: RecoveryWakeup })
|
||||
? "recovery.reconcile_successful_run_handoff_missing_state"
|
||||
: input.recoveryCause === "workspace_validation_failed"
|
||||
? "recovery.reconcile_workspace_validation_failed"
|
||||
: input.recoveryCause === "configuration_incomplete"
|
||||
? "recovery.reconcile_configuration_incomplete"
|
||||
: "recovery.reconcile_stranded_assigned_issue",
|
||||
recoveryCause: input.recoveryCause ?? "stranded_assigned_issue",
|
||||
latestRunId: input.latestRun?.id ?? null,
|
||||
|
||||
@@ -203,6 +203,15 @@ export type RuntimeSecretManifestEntry = {
|
||||
errorCode?: string | null;
|
||||
};
|
||||
|
||||
export type MissingRuntimeBinding = {
|
||||
consumerType: SecretBindingTargetType;
|
||||
consumerId: string;
|
||||
configPath: string;
|
||||
envKey: string;
|
||||
secretId: string;
|
||||
secretName: string | null;
|
||||
};
|
||||
|
||||
type RuntimeSecretResolution = {
|
||||
value: string;
|
||||
manifestEntry: RuntimeSecretManifestEntry;
|
||||
@@ -2352,6 +2361,60 @@ export function secretService(db: Db) {
|
||||
return { env: resolved, secretKeys, manifest };
|
||||
},
|
||||
|
||||
// Pre-dispatch validation: list declared secret refs in an env-like config
|
||||
// that have no binding for the given consumer, WITHOUT resolving any secret
|
||||
// values. Callers use this to surface a configuration-incomplete blocker
|
||||
// before a run is dispatched instead of letting resolution throw mid-setup.
|
||||
collectMissingRuntimeBindings: async (
|
||||
companyId: string,
|
||||
envValue: unknown,
|
||||
context: Omit<SecretConsumerContext, "configPath">,
|
||||
): Promise<MissingRuntimeBinding[]> => {
|
||||
const record = asRecord(envValue);
|
||||
if (!record) return [];
|
||||
const secretRefs = Object.entries(record).flatMap(([key, rawBinding]) => {
|
||||
if (!ENV_KEY_RE.test(key)) return [];
|
||||
const parsed = envBindingSchema.safeParse(rawBinding);
|
||||
if (!parsed.success) return [];
|
||||
const binding = canonicalizeBinding(parsed.data as EnvBinding);
|
||||
if (binding.type !== "secret_ref") return [];
|
||||
return [{ key, configPath: `env.${key}`, secretId: binding.secretId }];
|
||||
});
|
||||
if (secretRefs.length === 0) return [];
|
||||
|
||||
const bindingChecks = await Promise.all(secretRefs.map(async (entry) => ({
|
||||
entry,
|
||||
found: await getBinding({
|
||||
companyId,
|
||||
secretId: entry.secretId,
|
||||
consumerType: context.consumerType,
|
||||
consumerId: context.consumerId,
|
||||
configPath: entry.configPath,
|
||||
}),
|
||||
})));
|
||||
const missingEntries = bindingChecks
|
||||
.filter((check) => !check.found)
|
||||
.map((check) => check.entry);
|
||||
if (missingEntries.length === 0) return [];
|
||||
|
||||
const secretRows = await Promise.all(
|
||||
[...new Set(missingEntries.map((entry) => entry.secretId))].map(async (secretId) => [
|
||||
secretId,
|
||||
await getById(secretId).catch(() => null),
|
||||
] as const),
|
||||
);
|
||||
const secretsById = new Map(secretRows);
|
||||
|
||||
return missingEntries.map((entry) => ({
|
||||
consumerType: context.consumerType,
|
||||
consumerId: context.consumerId,
|
||||
configPath: entry.configPath,
|
||||
envKey: entry.key,
|
||||
secretId: entry.secretId,
|
||||
secretName: secretsById.get(entry.secretId)?.name ?? null,
|
||||
}));
|
||||
},
|
||||
|
||||
resolveAdapterConfigForRuntime: async (
|
||||
companyId: string,
|
||||
adapterConfig: Record<string, unknown>,
|
||||
|
||||
Reference in New Issue
Block a user