feat(adapter-utils): add observable sandbox sync progress (#8395)
## Thinking Path > - Paperclip is the control plane for running AI-agent companies, so long-running remote work needs to stay observable to human operators. > - Cloud / sandbox agents are an active roadmap area, and their workspace sync path is part of the runtime substrate every remote coding run depends on. > - In the sandbox and SSH execution-target flows, Paperclip logged that sync had started, then often went silent for the full transfer window. > - That made large remote syncs feel stalled and also hid a real performance problem in the command-managed sandbox upload path. > - The first part of this pull request threads a throttled progress-reporting surface through the adapter execution-target stack so sync and restore work can emit meaningful updates. > - The second part fixes the command-managed sandbox transport itself: it removes the old serial 32KB append bottleneck, but also falls back away from the single-stream path when a provider-backed sandbox runner cannot surface mid-flight stdin progress. > - The result is that sandbox and SSH transfers are both faster and more observable, including the live Daytona-style sandbox case that previously only emitted `0%` and `100%`. ## Linked Issues or Issue Description No public GitHub issue exists for this bug, so it is described inline below following the bug report template. ### What happened - Remote sandbox and SSH workspace syncs could spend a long time transferring data while only logging a start line (`Syncing workspace and runtime assets to sandbox environment`) and, at best, a terminal line. - In the command-managed sandbox path, the original upload implementation also paid a large performance cost by appending base64 data in many small sequential remote writes (thousands of serial 32KB round-trips on a large workspace). - After the initial transport rewrite, live provider-backed sandbox runs still only emitted `0%` and `100%` because the single-stream stdin RPC buffered progress until completion. ### Expected behavior - Long-running sandbox and SSH syncs should periodically report how much of the transfer is complete (a percentage and/or MB transferred) so an operator can tell the run is healthy and making progress rather than stuck. - The main sandbox upload path should not be artificially slow. - A transfer that fails partway should leave an explicit failure marker in the log rather than a dangling intermediate percentage. ### Steps to reproduce 1. Run an agent against a sandbox (command-managed) or SSH (remote-managed) execution target with a non-trivial workspace. 2. Watch the run log during the workspace/runtime asset sync phase. 3. Observe that the log shows the sync start line and then stays silent for the full transfer (live provider-backed sandbox runs only show `0%` then `100%`). ### Paperclip version or commit - Branch `PAPA-825-provide-status-updates-when-syncing-sandboxes` off `master`. ### Deployment mode - Self-hosted / local instance using sandbox (command-managed) and SSH (remote-managed) execution targets, including provider-backed sandbox runners. ## What Changed - Added shared throttled runtime progress reporting and threaded `onProgress` through the adapter execution-target surface and adapter `execute.ts` entrypoints. - Added sync and restore progress reporting for the command-managed sandbox path and the SSH/remote-managed path, including git import/export progress where totals are known. - Reworked command-managed sandbox transfer behavior so uploads use the faster single-stream path when appropriate, but fall back to chunked progress-emitting writes when the runner cannot expose mid-stream stdin progress. - Marked provider-backed environment sandbox runners as not supporting single-stream stdin progress so live sandbox runs emit meaningful intermediate updates instead of only `0%` and `100%`. - Emit an explicit terminal failure marker (`failed at NN% (x/y MB)`) when an SSH/tar transfer rejects, so a failed sync no longer leaves a dangling intermediate percentage in the log. - Run the SSH sync/restore size estimate (local directory walk / remote `du` probe) concurrently with the transfer instead of awaiting it before opening the pipe, so progress instrumentation no longer adds startup latency proportional to workspace file count. - Added and extended focused regression coverage for runtime progress throttling and the new failure marker, command-managed sandbox transfers, sandbox orchestration, SSH transfer progress, and environment execution-target wiring. ## Verification - `pnpm exec vitest run packages/adapter-utils/src/runtime-progress.test.ts packages/adapter-utils/src/ssh-fixture.test.ts packages/adapter-utils/src/command-managed-runtime.test.ts packages/adapter-utils/src/sandbox-managed-runtime.test.ts` - `pnpm exec vitest run packages/adapter-utils/src/command-managed-runtime.test.ts server/src/__tests__/environment-execution-target.test.ts` - `npx tsc --noEmit` for `packages/adapter-utils` ## Risks - The provider-backed sandbox fallback now prefers chunked command-managed writes when progress hooks are active, so small-to-medium uploads may trade some raw throughput for observable intermediate progress on runtimes that cannot surface true mid-stream stdin progress. - Progress percentages on tar-based transfers still depend on estimates in some cases, so operators may briefly see MB-only lines before the estimate resolves, then near-final clamping before the terminal `100%` line. - This PR changes shared execution-target behavior used by multiple adapters, so regressions would most likely appear in remote runtime setup/teardown flows rather than in a single adapter. ## Model Used - Initial implementation: OpenAI GPT-5.4 via Codex local agent (`codex_local`), high reasoning mode. - Observability follow-ups (failure marker, concurrent size estimate, added tests): Claude Opus 4.8 via Claude Code (`claude_local`). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
@@ -8,8 +8,16 @@ import {
|
||||
} from "./sandbox-managed-runtime.js";
|
||||
import { preferredShellForSandbox, shellCommandArgs } from "./sandbox-shell.js";
|
||||
import type { RunProcessResult } from "./server-utils.js";
|
||||
import type { RuntimeProgressSink } from "./runtime-progress.js";
|
||||
|
||||
export interface CommandManagedRuntimeRunner {
|
||||
/**
|
||||
* True only when `execute({ stdin })` can surface useful in-flight progress
|
||||
* for a single stdin-backed command. Provider-backed sandbox runners usually
|
||||
* complete the entire RPC before returning, so they should leave this false
|
||||
* and let the caller choose a chunked upload path when progress is requested.
|
||||
*/
|
||||
supportsSingleStreamStdinProgress?: boolean;
|
||||
execute(input: {
|
||||
command: string;
|
||||
args?: string[];
|
||||
@@ -40,7 +48,14 @@ function mergeRuntimeExcludes(entries: string[] | undefined): string[] {
|
||||
return [...new Set([".paperclip-runtime", ...(entries ?? [])])];
|
||||
}
|
||||
|
||||
const REMOTE_WRITE_BASE64_CHUNK_SIZE = 32 * 1024;
|
||||
// Largest base64 body we hand to the runner as a single stdin string. Normal
|
||||
// multi-MB workspace/asset tarballs stay well under this and upload in one
|
||||
// round-trip; anything larger uses the bounded chunked-append fallback so a
|
||||
// runaway stdin string can't blow the runner/provider RPC limits.
|
||||
const REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES = 96 * 1024 * 1024;
|
||||
// Fallback chunk size (base64 bytes). Kept a multiple of 4 so each chunk is a
|
||||
// self-contained base64 unit that decodes cleanly on its own.
|
||||
const REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE = 4 * 1024 * 1024;
|
||||
|
||||
function toBuffer(bytes: Buffer | Uint8Array | ArrayBuffer): Buffer {
|
||||
if (Buffer.isBuffer(bytes)) return bytes;
|
||||
@@ -62,13 +77,21 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
shellCommand?: "bash" | "sh" | null;
|
||||
}): SandboxManagedRuntimeClient {
|
||||
const shellCommand = preferredShellForSandbox(input.shellCommand);
|
||||
const runShell = async (script: string, opts: { stdin?: string; timeoutMs?: number } = {}) => {
|
||||
const runShell = async (
|
||||
script: string,
|
||||
opts: {
|
||||
stdin?: string;
|
||||
timeoutMs?: number;
|
||||
onLog?: (stream: "stdout" | "stderr", chunk: string) => Promise<void>;
|
||||
} = {},
|
||||
) => {
|
||||
const result = await input.runner.execute({
|
||||
command: shellCommand,
|
||||
args: shellCommandArgs(script),
|
||||
cwd: input.commandCwd,
|
||||
stdin: opts.stdin,
|
||||
timeoutMs: opts.timeoutMs ?? input.timeoutMs,
|
||||
onLog: opts.onLog,
|
||||
});
|
||||
requireSuccessfulResult(result, script);
|
||||
return result;
|
||||
@@ -78,25 +101,101 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
makeDir: async (remotePath) => {
|
||||
await runShell(`mkdir -p ${shellQuote(remotePath)}`);
|
||||
},
|
||||
writeFile: async (remotePath, bytes) => {
|
||||
const body = toBuffer(bytes).toString("base64");
|
||||
writeFile: async (remotePath, bytes, options) => {
|
||||
const buffer = toBuffer(bytes);
|
||||
const total = buffer.byteLength;
|
||||
const body = buffer.toString("base64");
|
||||
const remoteDir = path.posix.dirname(remotePath);
|
||||
const remoteTempPath = `${remotePath}.paperclip-upload.b64`;
|
||||
const remoteTempPath = `${remotePath}.paperclip-upload`;
|
||||
const canUseSingleStreamProgressPath =
|
||||
!options?.onProgress || input.runner.supportsSingleStreamStdinProgress === true;
|
||||
|
||||
await runShell(
|
||||
`mkdir -p ${shellQuote(remoteDir)} && rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`,
|
||||
);
|
||||
for (let offset = 0; offset < body.length; offset += REMOTE_WRITE_BASE64_CHUNK_SIZE) {
|
||||
const chunk = body.slice(offset, offset + REMOTE_WRITE_BASE64_CHUNK_SIZE);
|
||||
await runShell(`printf '%s' ${shellQuote(chunk)} >> ${shellQuote(remoteTempPath)}`);
|
||||
// Primary path: a single round-trip. Stream the entire base64 body to one
|
||||
// `base64 -d` process via stdin, decode straight into a temp file, then
|
||||
// atomically rename into place. This replaces the previous loop that did
|
||||
// one `printf >> tmpfile` shell round-trip per 32 KB — thousands of serial
|
||||
// processes for a large workspace — with exactly one process.
|
||||
if (
|
||||
body.length <= REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES &&
|
||||
canUseSingleStreamProgressPath
|
||||
) {
|
||||
await options?.onProgress?.(0, total);
|
||||
await runShell(
|
||||
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
||||
`base64 -d > ${shellQuote(remoteTempPath)} && ` +
|
||||
`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`,
|
||||
{ stdin: body },
|
||||
);
|
||||
await options?.onProgress?.(total, total);
|
||||
return;
|
||||
}
|
||||
|
||||
// Bounded fallback for payloads too large to hand the runner as one stdin
|
||||
// string: append the base64 body to a remote temp file in large chunks
|
||||
// (orders of magnitude fewer round-trips than the old 32 KB loop), decoding
|
||||
// each self-contained chunk on arrival and emitting progress per write,
|
||||
// then atomically rename into place.
|
||||
await runShell(
|
||||
`base64 -d < ${shellQuote(remoteTempPath)} > ${shellQuote(remotePath)} && rm -f ${shellQuote(remoteTempPath)}`,
|
||||
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
||||
`rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`,
|
||||
);
|
||||
for (let offset = 0; offset < body.length; offset += REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE) {
|
||||
const chunk = body.slice(offset, offset + REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE);
|
||||
await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk });
|
||||
const decodedSoFar = Math.min(total, Math.floor(((offset + chunk.length) * 3) / 4));
|
||||
await options?.onProgress?.(decodedSoFar, total);
|
||||
}
|
||||
await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`);
|
||||
await options?.onProgress?.(total, total);
|
||||
},
|
||||
readFile: async (remotePath) => {
|
||||
const result = await runShell(`base64 < ${shellQuote(remotePath)}`);
|
||||
return Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
|
||||
readFile: async (remotePath, options) => {
|
||||
// Decoded file size up front so download progress can be a percentage.
|
||||
// Only paid when a progress hook is attached.
|
||||
let totalBytes: number | null = null;
|
||||
if (options?.onProgress) {
|
||||
const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`);
|
||||
const parsed = Number.parseInt(sizeResult.stdout.trim(), 10);
|
||||
totalBytes = Number.isFinite(parsed) && parsed >= 0 ? parsed : null;
|
||||
}
|
||||
|
||||
// Stream the remote `base64` stdout through a byte counter, decoding each
|
||||
// 4-char-aligned slice incrementally rather than buffering the whole
|
||||
// response as one string. Falls back to the buffered result when the
|
||||
// runner does not surface incremental stdout.
|
||||
const decodedChunks: Buffer[] = [];
|
||||
let b64Remainder = "";
|
||||
let decodedSoFar = 0;
|
||||
let streamed = false;
|
||||
const result = await runShell(`base64 < ${shellQuote(remotePath)}`, {
|
||||
onLog: async (stream, chunk) => {
|
||||
if (stream !== "stdout") return;
|
||||
streamed = true;
|
||||
const data = b64Remainder + chunk.replace(/\s+/g, "");
|
||||
const alignedLen = data.length - (data.length % 4);
|
||||
if (alignedLen > 0) {
|
||||
const decoded = Buffer.from(data.slice(0, alignedLen), "base64");
|
||||
decodedChunks.push(decoded);
|
||||
decodedSoFar += decoded.byteLength;
|
||||
}
|
||||
b64Remainder = data.slice(alignedLen);
|
||||
if (options?.onProgress) {
|
||||
const done = totalBytes != null ? Math.min(totalBytes, decodedSoFar) : decodedSoFar;
|
||||
await options.onProgress(done, totalBytes);
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
let out: Buffer;
|
||||
if (streamed) {
|
||||
if (b64Remainder.length > 0) {
|
||||
decodedChunks.push(Buffer.from(b64Remainder, "base64"));
|
||||
}
|
||||
out = Buffer.concat(decodedChunks);
|
||||
} else {
|
||||
out = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
|
||||
}
|
||||
await options?.onProgress?.(out.byteLength, totalBytes ?? out.byteLength);
|
||||
return out;
|
||||
},
|
||||
listFiles: async (remotePath) => {
|
||||
const result = await runShell(
|
||||
@@ -146,6 +245,9 @@ export async function prepareCommandManagedRuntime(input: {
|
||||
installCommand?: string | null;
|
||||
/** When provided alongside `installCommand`, skip the install if `command -v <detectCommand>` succeeds. */
|
||||
detectCommand?: string | null;
|
||||
// Upload progress sink. Forwarded to prepareSandboxManagedRuntime; the child
|
||||
// task wires it into the byte-counting writeFile/readFile transport.
|
||||
onProgress?: RuntimeProgressSink;
|
||||
}): Promise<PreparedSandboxManagedRuntime> {
|
||||
const timeoutMs = input.spec.timeoutMs && input.spec.timeoutMs > 0 ? input.spec.timeoutMs : 300_000;
|
||||
const workspaceRemoteDir = input.workspaceRemoteDir ?? input.spec.remoteCwd;
|
||||
@@ -193,6 +295,7 @@ export async function prepareCommandManagedRuntime(input: {
|
||||
workspaceExclude: mergeRuntimeExcludes(input.workspaceExclude),
|
||||
preserveAbsentOnRestore: input.preserveAbsentOnRestore,
|
||||
assets: input.assets,
|
||||
onProgress: input.onProgress,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -227,5 +330,6 @@ export async function prepareCommandManagedRuntime(input: {
|
||||
workspaceExclude: mergeRuntimeExcludes(input.workspaceExclude),
|
||||
preserveAbsentOnRestore: input.preserveAbsentOnRestore,
|
||||
assets: input.assets,
|
||||
onProgress: input.onProgress,
|
||||
});
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user