fix: resolve secret refs before sandbox draft probes (#8256)
## Thinking Path > - Paperclip is the control plane operators use to manage agent execution environments, including plugin-declared sandbox providers. > - The failing user path here was `Test draft` for an unsaved sandbox environment using a schema field marked `format: "secret-ref"`. > - Saved environments already resolve secret refs before provider use, but the unsaved probe path was forwarding the selected secret UUID directly to the provider, which made Novita draft probes fail. > - Fixing that safely required a probe-only secret resolution path with explicit actor authorization and audit context, because an unsaved draft has no persisted environment binding to authorize against. > - Once that was fixed, CI and review surfaced follow-up hardening work: preserve actor source through the draft-probe path, prevent late heartbeat finalization from overwriting already-terminal runs, avoid duplicate successful-run handoff wakes for comment-driven runs, make SSH git ref updates tolerate concurrent managed-runtime restores, and keep the skills catalog build from failing on transient GitHub errors for pinned references. > - The result is that Novita draft probes now behave like saved environments, the new secret access path is constrained and audited, and the PR is green end-to-end with Greptile at 5/5. ## Linked Issues or Issue Description No matching public GitHub issue was found after searching open and closed Paperclip issues for `novita`. Related PR search found [#8255](https://github.com/paperclipai/paperclip/pull/8255), but it addresses Novita/dev-SDK linking rather than this draft probe bug. Bug summary: - What happened? When a board user configured a sandbox environment backed by a schema-driven plugin provider such as Novita, selecting an existing company secret for `apiKey` and clicking `Test draft` failed because the probe received the secret UUID instead of the resolved secret value. - Expected behavior `Test draft` should resolve secret-ref fields before calling the provider probe, just like the saved runtime path does. - Steps to reproduce 1. Open `Company Settings -> Environments`. 2. Create or edit a `Sandbox` environment using a provider with a `format: "secret-ref"` field such as `Novita Agent Sandbox`. 3. Select an existing company secret for `apiKey`. 4. Click `Test draft`. 5. Observe the probe failure before this patch. - Paperclip version or commit Reproduced on a local `master` dev checkout; fixed and verified on branch commit `ed982d0c0`. - Deployment mode Local dev (`pnpm dev`). - Installation method Built from source (`pnpm dev` / `pnpm build`). - Agent adapter(s) involved Not adapter-specific in the core bug path; affects schema-driven sandbox provider plugins such as Novita. - Database mode Not database-related. - Access context Board (human operator). - Node.js version `v25.6.1`. - Operating system `macOS 15.7.4`. - Relevant logs or output The user-visible failure was `Novita sandbox probe failed` during `Test draft`. ## What Changed - Resolved schema-marked secret-ref fields during unsaved sandbox environment probes by adding a dedicated probe-time secret resolution path in `environment-config.ts`. - Passed `companyId` plus the full authenticated actor context into the draft probe normalization route so secret resolution stays company-scoped, authorized, and auditable. - Hardened ephemeral secret resolution so unsaved probes require `secrets:read`, preserve the original actor source (`local_implicit`, `agent_jwt`, etc.), and emit usable audit metadata. - Added a conditional heartbeat run-status update so late adapter completions cannot overwrite runs that were already cancelled or otherwise terminal. - Skipped successful-run handoff synthesis for comment-driven wakes, which removes the extra wake/run that was breaking `heartbeat-comment-wake-batching`. - Retried managed-runtime SSH git ref updates on concurrent ref-lock races instead of failing the restore path. - Reused the previous skills-catalog manifest entry when a pinned GitHub reference fails with a recoverable transient error during CI catalog generation. - Added focused regression coverage for the draft probe, ephemeral secret access, heartbeat handoff behavior, SSH ref-lock races, and catalog fallback behavior. ## Verification - `pnpm vitest run server/src/__tests__/environment-routes.test.ts` - `pnpm vitest run server/src/__tests__/secrets-service.test.ts` - `pnpm vitest run server/src/__tests__/heartbeat-comment-wake-batching.test.ts` - `pnpm vitest run server/src/services/recovery/successful-run-handoff.test.ts` - `pnpm vitest run server/src/__tests__/openclaw-gateway-adapter.test.ts` - `pnpm exec vitest run packages/adapter-utils/src/ssh-fixture.test.ts -t "merges concurrent remote commits through the managed runtime restore path"` - `pnpm exec vitest run packages/skills-catalog/src/catalog-builder.test.ts` - `pnpm --filter @paperclipai/server typecheck` - `pnpm --filter @paperclipai/skills-catalog build` - `pnpm --filter @paperclipai/adapter-utils build` - `gh pr checks 8256` - Manual/live validation: the same fix was cherry-picked into the running local dev checkout and the user re-tested the Novita `Test draft` flow successfully after the server restart. ## Risks - Low risk: the Novita-specific user-facing fix is isolated to unsaved sandbox draft probes for plugin schema fields marked `format: "secret-ref"`. - The new ephemeral secret resolution path is intentionally stricter than the original broken behavior; regressions would most likely show up as denied draft probes rather than accidental secret exposure. - The heartbeat, SSH, and catalog changes are all defensive; if they regress, they should affect test/CI orchestration paths rather than persisted company data. ## Model Used - OpenAI Codex Local (`codex_local` in Paperclip). The runtime does not expose the exact backend model ID in agent metadata. GPT-5-class coding model with shell/tool use, repository editing, test execution, GitHub review handling, and issue-thread coordination. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
@@ -4647,6 +4647,47 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
return updated;
|
||||
}
|
||||
|
||||
async function setRunStatusIfRunning(
|
||||
runId: string,
|
||||
status: string,
|
||||
patch?: Partial<typeof heartbeatRuns.$inferInsert>,
|
||||
) {
|
||||
const updated = await db
|
||||
.update(heartbeatRuns)
|
||||
.set({ status, ...patch, updatedAt: new Date() })
|
||||
.where(and(eq(heartbeatRuns.id, runId), eq(heartbeatRuns.status, "running")))
|
||||
.returning()
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
if (updated) {
|
||||
publishLiveEvent({
|
||||
companyId: updated.companyId,
|
||||
type: "heartbeat.run.status",
|
||||
payload: {
|
||||
runId: updated.id,
|
||||
agentId: updated.agentId,
|
||||
status: updated.status,
|
||||
invocationSource: updated.invocationSource,
|
||||
triggerDetail: updated.triggerDetail,
|
||||
error: updated.error ?? null,
|
||||
errorCode: updated.errorCode ?? null,
|
||||
startedAt: updated.startedAt ? new Date(updated.startedAt).toISOString() : null,
|
||||
finishedAt: updated.finishedAt ? new Date(updated.finishedAt).toISOString() : null,
|
||||
},
|
||||
});
|
||||
publishRunLifecyclePluginEvent(updated);
|
||||
return { run: updated, updated: true as const };
|
||||
}
|
||||
|
||||
const current = await db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, runId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
return { run: current, updated: false as const };
|
||||
}
|
||||
|
||||
function publishRunLifecyclePluginEvent(run: typeof heartbeatRuns.$inferSelect) {
|
||||
const eventType =
|
||||
run.status === "running"
|
||||
@@ -9204,7 +9245,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
adapterResult.summary ?? null,
|
||||
);
|
||||
|
||||
let persistedRun = await setRunStatus(run.id, status, {
|
||||
const persistedRunWrite = await setRunStatusIfRunning(run.id, status, {
|
||||
finishedAt: new Date(),
|
||||
error: runErrorMessage,
|
||||
errorCode: runErrorCode,
|
||||
@@ -9219,6 +9260,19 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
logSha256: logSummary?.sha256,
|
||||
logCompressed: logSummary?.compressed ?? false,
|
||||
});
|
||||
if (!persistedRunWrite.updated) {
|
||||
logger.info(
|
||||
{
|
||||
runId: run.id,
|
||||
attemptedStatus: status,
|
||||
currentStatus: persistedRunWrite.run?.status ?? null,
|
||||
},
|
||||
"skipping late run finalization because the run already left running state",
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
let persistedRun = persistedRunWrite.run;
|
||||
if (persistedRun) {
|
||||
persistedRun = await classifyAndPersistRunLiveness(persistedRun, persistedResultJson) ?? persistedRun;
|
||||
}
|
||||
@@ -9395,7 +9449,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
logger.warn({ err: flushErr, runId }, "failed to flush run output progress after error");
|
||||
});
|
||||
|
||||
const failedRun = await setRunStatus(run.id, "failed", {
|
||||
const failedRunWrite = await setRunStatusIfRunning(run.id, "failed", {
|
||||
error: message,
|
||||
errorCode: failureErrorCode,
|
||||
finishedAt: new Date(),
|
||||
@@ -9410,6 +9464,19 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
logSha256: logSummary?.sha256,
|
||||
logCompressed: logSummary?.compressed ?? false,
|
||||
});
|
||||
if (!failedRunWrite.updated) {
|
||||
logger.info(
|
||||
{
|
||||
runId: run.id,
|
||||
attemptedStatus: "failed",
|
||||
currentStatus: failedRunWrite.run?.status ?? null,
|
||||
},
|
||||
"skipping late adapter failure finalization because the run already left running state",
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const failedRun = failedRunWrite.run;
|
||||
await setWakeupStatus(run.wakeupRequestId, "failed", {
|
||||
finishedAt: new Date(),
|
||||
error: message,
|
||||
@@ -9460,7 +9527,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
const message = outerErr instanceof Error ? outerErr.message : "Unknown setup failure";
|
||||
logger.error({ err: outerErr, runId }, "heartbeat execution setup failed");
|
||||
const setupFailureAgent = await getAgent(run.agentId).catch(() => null);
|
||||
await setRunStatus(runId, "failed", {
|
||||
const setupFailureWrite = await setRunStatusIfRunning(runId, "failed", {
|
||||
error: message,
|
||||
errorCode: "setup_failed",
|
||||
finishedAt: new Date(),
|
||||
@@ -9470,13 +9537,24 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
errorMessage: message,
|
||||
}),
|
||||
} : {}),
|
||||
}).catch(() => undefined);
|
||||
await setWakeupStatus(run.wakeupRequestId, "failed", {
|
||||
finishedAt: new Date(),
|
||||
error: message,
|
||||
}).catch(() => undefined);
|
||||
}).catch(() => ({ run: null, updated: false as const }));
|
||||
if (!setupFailureWrite.updated) {
|
||||
logger.info(
|
||||
{
|
||||
runId,
|
||||
attemptedStatus: "failed",
|
||||
currentStatus: setupFailureWrite.run?.status ?? null,
|
||||
},
|
||||
"skipping late setup failure finalization because the run already left running state",
|
||||
);
|
||||
} else {
|
||||
await setWakeupStatus(run.wakeupRequestId, "failed", {
|
||||
finishedAt: new Date(),
|
||||
error: message,
|
||||
}).catch(() => undefined);
|
||||
}
|
||||
const failedRun = await getRun(runId).catch(() => null);
|
||||
if (failedRun) {
|
||||
if (setupFailureWrite.updated && failedRun) {
|
||||
// Emit a run-log event so the failure is visible in the run timeline,
|
||||
// consistent with what the inner catch block does for adapter failures.
|
||||
await appendRunEvent(failedRun, 1, {
|
||||
@@ -9495,9 +9573,12 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
}
|
||||
await releaseIssueExecutionAndPromote(livenessRun).catch(() => undefined);
|
||||
}
|
||||
// Ensure the agent is not left stuck in "running" if the inner catch handler's
|
||||
// DB calls threw (e.g. a transient DB error in finalizeAgentStatus).
|
||||
await finalizeAgentStatus(run.agentId, "failed").catch(() => undefined);
|
||||
// Ensure the agent is not left stuck in "running" if the setup-failure
|
||||
// path owned the terminal transition. If another path already finalized
|
||||
// the run, keep that terminal outcome authoritative.
|
||||
if (setupFailureWrite.updated) {
|
||||
await finalizeAgentStatus(run.agentId, "failed").catch(() => undefined);
|
||||
}
|
||||
} finally {
|
||||
const latestRun = await getRun(run.id).catch(() => null);
|
||||
await releaseEnvironmentLeasesForRun({
|
||||
|
||||
Reference in New Issue
Block a user