feat(adapter-utils): add observable sandbox sync progress (#8395)
## Thinking Path > - Paperclip is the control plane for running AI-agent companies, so long-running remote work needs to stay observable to human operators. > - Cloud / sandbox agents are an active roadmap area, and their workspace sync path is part of the runtime substrate every remote coding run depends on. > - In the sandbox and SSH execution-target flows, Paperclip logged that sync had started, then often went silent for the full transfer window. > - That made large remote syncs feel stalled and also hid a real performance problem in the command-managed sandbox upload path. > - The first part of this pull request threads a throttled progress-reporting surface through the adapter execution-target stack so sync and restore work can emit meaningful updates. > - The second part fixes the command-managed sandbox transport itself: it removes the old serial 32KB append bottleneck, but also falls back away from the single-stream path when a provider-backed sandbox runner cannot surface mid-flight stdin progress. > - The result is that sandbox and SSH transfers are both faster and more observable, including the live Daytona-style sandbox case that previously only emitted `0%` and `100%`. ## Linked Issues or Issue Description No public GitHub issue exists for this bug, so it is described inline below following the bug report template. ### What happened - Remote sandbox and SSH workspace syncs could spend a long time transferring data while only logging a start line (`Syncing workspace and runtime assets to sandbox environment`) and, at best, a terminal line. - In the command-managed sandbox path, the original upload implementation also paid a large performance cost by appending base64 data in many small sequential remote writes (thousands of serial 32KB round-trips on a large workspace). - After the initial transport rewrite, live provider-backed sandbox runs still only emitted `0%` and `100%` because the single-stream stdin RPC buffered progress until completion. ### Expected behavior - Long-running sandbox and SSH syncs should periodically report how much of the transfer is complete (a percentage and/or MB transferred) so an operator can tell the run is healthy and making progress rather than stuck. - The main sandbox upload path should not be artificially slow. - A transfer that fails partway should leave an explicit failure marker in the log rather than a dangling intermediate percentage. ### Steps to reproduce 1. Run an agent against a sandbox (command-managed) or SSH (remote-managed) execution target with a non-trivial workspace. 2. Watch the run log during the workspace/runtime asset sync phase. 3. Observe that the log shows the sync start line and then stays silent for the full transfer (live provider-backed sandbox runs only show `0%` then `100%`). ### Paperclip version or commit - Branch `PAPA-825-provide-status-updates-when-syncing-sandboxes` off `master`. ### Deployment mode - Self-hosted / local instance using sandbox (command-managed) and SSH (remote-managed) execution targets, including provider-backed sandbox runners. ## What Changed - Added shared throttled runtime progress reporting and threaded `onProgress` through the adapter execution-target surface and adapter `execute.ts` entrypoints. - Added sync and restore progress reporting for the command-managed sandbox path and the SSH/remote-managed path, including git import/export progress where totals are known. - Reworked command-managed sandbox transfer behavior so uploads use the faster single-stream path when appropriate, but fall back to chunked progress-emitting writes when the runner cannot expose mid-stream stdin progress. - Marked provider-backed environment sandbox runners as not supporting single-stream stdin progress so live sandbox runs emit meaningful intermediate updates instead of only `0%` and `100%`. - Emit an explicit terminal failure marker (`failed at NN% (x/y MB)`) when an SSH/tar transfer rejects, so a failed sync no longer leaves a dangling intermediate percentage in the log. - Run the SSH sync/restore size estimate (local directory walk / remote `du` probe) concurrently with the transfer instead of awaiting it before opening the pipe, so progress instrumentation no longer adds startup latency proportional to workspace file count. - Added and extended focused regression coverage for runtime progress throttling and the new failure marker, command-managed sandbox transfers, sandbox orchestration, SSH transfer progress, and environment execution-target wiring. ## Verification - `pnpm exec vitest run packages/adapter-utils/src/runtime-progress.test.ts packages/adapter-utils/src/ssh-fixture.test.ts packages/adapter-utils/src/command-managed-runtime.test.ts packages/adapter-utils/src/sandbox-managed-runtime.test.ts` - `pnpm exec vitest run packages/adapter-utils/src/command-managed-runtime.test.ts server/src/__tests__/environment-execution-target.test.ts` - `npx tsc --noEmit` for `packages/adapter-utils` ## Risks - The provider-backed sandbox fallback now prefers chunked command-managed writes when progress hooks are active, so small-to-medium uploads may trade some raw throughput for observable intermediate progress on runtimes that cannot surface true mid-stream stdin progress. - Progress percentages on tar-based transfers still depend on estimates in some cases, so operators may briefly see MB-only lines before the estimate resolves, then near-final clamping before the terminal `100%` line. - This PR changes shared execution-target behavior used by multiple adapters, so regressions would most likely appear in remote runtime setup/teardown flows rather than in a single adapter. ## Model Used - Initial implementation: OpenAI GPT-5.4 via Codex local agent (`codex_local`), high reasoning mode. - Observability follow-ups (failure marker, concurrent size estimate, added tests): Claude Opus 4.8 via Claude Code (`claude_local`). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
@@ -1,15 +1,80 @@
|
||||
import { mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { execFile as execFileCallback, spawn } from "node:child_process";
|
||||
import { promisify } from "node:util";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
|
||||
import { prepareCommandManagedRuntime } from "./command-managed-runtime.js";
|
||||
import {
|
||||
createCommandManagedRuntimeClient,
|
||||
prepareCommandManagedRuntime,
|
||||
type CommandManagedRuntimeRunner,
|
||||
} from "./command-managed-runtime.js";
|
||||
import type { RunProcessResult } from "./server-utils.js";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
|
||||
interface SpawnRunnerHandle {
|
||||
runner: CommandManagedRuntimeRunner;
|
||||
calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }>;
|
||||
}
|
||||
|
||||
// A runner that actually executes the shell scripts (piping stdin through a real
|
||||
// pipe so multi-MB payloads work) and replays stdout through onLog in several
|
||||
// chunks so the streaming readFile byte-counter is exercised.
|
||||
function makeSpawnRunner(options: { supportsSingleStreamStdinProgress?: boolean } = {}): SpawnRunnerHandle {
|
||||
const calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }> = [];
|
||||
const runner: CommandManagedRuntimeRunner = {
|
||||
supportsSingleStreamStdinProgress: options.supportsSingleStreamStdinProgress,
|
||||
execute: async (input) =>
|
||||
await new Promise<RunProcessResult>((resolve) => {
|
||||
calls.push({ command: input.command, args: input.args, cwd: input.cwd, stdin: input.stdin });
|
||||
const startedAt = new Date().toISOString();
|
||||
const command =
|
||||
input.command === "sh" ? "/bin/sh" : input.command === "bash" ? "/bin/bash" : input.command;
|
||||
const child = spawn(command, input.args ?? [], {
|
||||
cwd: input.cwd,
|
||||
env: { ...process.env, ...input.env },
|
||||
});
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
child.stdout.on("data", (chunk) => {
|
||||
stdout += chunk.toString("utf8");
|
||||
});
|
||||
child.stderr.on("data", (chunk) => {
|
||||
stderr += chunk.toString("utf8");
|
||||
});
|
||||
child.on("error", () => {
|
||||
resolve({ exitCode: 127, signal: null, timedOut: false, stdout, stderr, pid: null, startedAt });
|
||||
});
|
||||
child.on("close", async (code) => {
|
||||
if (input.onLog && stdout.length > 0) {
|
||||
const chunkSize = Math.max(1, Math.ceil(stdout.length / 4));
|
||||
for (let offset = 0; offset < stdout.length; offset += chunkSize) {
|
||||
await input.onLog("stdout", stdout.slice(offset, offset + chunkSize));
|
||||
}
|
||||
}
|
||||
resolve({
|
||||
exitCode: code ?? 0,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
stdout,
|
||||
stderr,
|
||||
pid: child.pid ?? null,
|
||||
startedAt,
|
||||
});
|
||||
});
|
||||
if (input.stdin != null) child.stdin.write(input.stdin);
|
||||
child.stdin.end();
|
||||
}),
|
||||
};
|
||||
return { runner, calls };
|
||||
}
|
||||
|
||||
function toArrayBuffer(buffer: Buffer): ArrayBuffer {
|
||||
return Uint8Array.from(buffer).buffer;
|
||||
}
|
||||
|
||||
describe("command managed runtime", () => {
|
||||
const cleanupDirs: string[] = [];
|
||||
|
||||
@@ -117,7 +182,9 @@ describe("command managed runtime", () => {
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, "README.md"), "utf8")).resolves.toBe("local workspace\n");
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, ".paperclip-runtime", "state.json"), "utf8")).rejects
|
||||
.toMatchObject({ code: "ENOENT" });
|
||||
expect(calls.every((call) => call.stdin == null)).toBe(true);
|
||||
// The single-stream upload pipes the tarball through exactly one stdin-backed
|
||||
// process (the speed fix); nothing else streams stdin.
|
||||
expect(calls.filter((call) => call.stdin != null).length).toBe(1);
|
||||
|
||||
await mkdir(path.join(remoteWorkspaceDir, ".paperclip-runtime"), { recursive: true });
|
||||
await writeFile(path.join(remoteWorkspaceDir, "README.md"), "remote workspace\n", "utf8");
|
||||
@@ -129,7 +196,9 @@ describe("command managed runtime", () => {
|
||||
.toBe("{\"keep\":true}\n");
|
||||
await expect(readFile(path.join(localWorkspaceDir, ".paperclip-runtime", "remote-state.json"), "utf8")).rejects
|
||||
.toMatchObject({ code: "ENOENT" });
|
||||
expect(calls.every((call) => call.stdin == null)).toBe(true);
|
||||
// Restore streams the download through `base64`/onLog (no stdin), so the only
|
||||
// stdin-backed call remains the single upload from prepare.
|
||||
expect(calls.filter((call) => call.stdin != null).length).toBe(1);
|
||||
});
|
||||
|
||||
it("runs setup commands from a stable root cwd when staging into a nested remote workspace dir", async () => {
|
||||
@@ -143,64 +212,7 @@ describe("command managed runtime", () => {
|
||||
await mkdir(remoteBaseDir, { recursive: true });
|
||||
await writeFile(path.join(localWorkspaceDir, "README.md"), "local workspace\n", "utf8");
|
||||
|
||||
const calls: Array<{
|
||||
command: string;
|
||||
args?: string[];
|
||||
cwd?: string;
|
||||
env?: Record<string, string>;
|
||||
stdin?: string;
|
||||
timeoutMs?: number;
|
||||
}> = [];
|
||||
const runner = {
|
||||
execute: async (input: {
|
||||
command: string;
|
||||
args?: string[];
|
||||
cwd?: string;
|
||||
env?: Record<string, string>;
|
||||
stdin?: string;
|
||||
timeoutMs?: number;
|
||||
}): Promise<RunProcessResult> => {
|
||||
calls.push({ ...input });
|
||||
const startedAt = new Date().toISOString();
|
||||
try {
|
||||
const result = await execFile(input.command === "sh" ? "/bin/sh" : input.command, input.args ?? [], {
|
||||
cwd: input.cwd,
|
||||
env: {
|
||||
...process.env,
|
||||
...input.env,
|
||||
},
|
||||
maxBuffer: 32 * 1024 * 1024,
|
||||
timeout: input.timeoutMs,
|
||||
});
|
||||
return {
|
||||
exitCode: 0,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
stdout: result.stdout,
|
||||
stderr: result.stderr,
|
||||
pid: null,
|
||||
startedAt,
|
||||
};
|
||||
} catch (error) {
|
||||
const err = error as NodeJS.ErrnoException & {
|
||||
stdout?: string;
|
||||
stderr?: string;
|
||||
code?: string | number | null;
|
||||
signal?: NodeJS.Signals | null;
|
||||
killed?: boolean;
|
||||
};
|
||||
return {
|
||||
exitCode: typeof err.code === "number" ? err.code : null,
|
||||
signal: err.signal ?? null,
|
||||
timedOut: Boolean(err.killed && input.timeoutMs),
|
||||
stdout: err.stdout ?? "",
|
||||
stderr: err.stderr ?? "",
|
||||
pid: null,
|
||||
startedAt,
|
||||
};
|
||||
}
|
||||
},
|
||||
};
|
||||
const { runner, calls } = makeSpawnRunner();
|
||||
|
||||
await prepareCommandManagedRuntime({
|
||||
runner,
|
||||
@@ -217,4 +229,99 @@ describe("command managed runtime", () => {
|
||||
expect(calls.every((call) => call.cwd === "/")).toBe(true);
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, "README.md"), "utf8")).resolves.toBe("local workspace\n");
|
||||
});
|
||||
|
||||
it("uploads a multi-MB payload in a single process and preserves exact bytes", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-write-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const remotePath = path.join(rootDir, "nested", "payload.bin");
|
||||
|
||||
// ~3 MB of every byte value so the test catches any non-binary-safe handling.
|
||||
const payload = Buffer.alloc(3 * 1024 * 1024);
|
||||
for (let i = 0; i < payload.length; i++) payload[i] = i % 256;
|
||||
|
||||
const { runner, calls } = makeSpawnRunner({ supportsSingleStreamStdinProgress: true });
|
||||
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
||||
|
||||
const progress: Array<{ done: number; total: number | null }> = [];
|
||||
await client.writeFile(remotePath, toArrayBuffer(payload), {
|
||||
onProgress: (done, total) => {
|
||||
progress.push({ done, total });
|
||||
},
|
||||
});
|
||||
|
||||
// Exactly one upload process: O(1) round-trips regardless of payload size.
|
||||
expect(calls.length).toBe(1);
|
||||
expect(calls[0].stdin).toBeTypeOf("string");
|
||||
|
||||
const written = await readFile(remotePath);
|
||||
expect(written.equals(payload)).toBe(true);
|
||||
|
||||
// Progress is monotonically non-decreasing and reaches the total.
|
||||
expect(progress.length).toBeGreaterThan(0);
|
||||
for (let i = 1; i < progress.length; i++) {
|
||||
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
|
||||
}
|
||||
expect(progress.at(-1)).toEqual({ done: payload.length, total: payload.length });
|
||||
});
|
||||
|
||||
it("falls back to chunked upload progress when the runner cannot report mid-stream stdin progress", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-write-fallback-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const remotePath = path.join(rootDir, "nested", "payload.bin");
|
||||
|
||||
const payload = Buffer.alloc(12 * 1024 * 1024);
|
||||
for (let i = 0; i < payload.length; i++) payload[i] = i % 256;
|
||||
|
||||
const { runner, calls } = makeSpawnRunner({ supportsSingleStreamStdinProgress: false });
|
||||
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
||||
|
||||
const progress: Array<{ done: number; total: number | null }> = [];
|
||||
await client.writeFile(remotePath, toArrayBuffer(payload), {
|
||||
onProgress: (done, total) => {
|
||||
progress.push({ done, total });
|
||||
},
|
||||
});
|
||||
|
||||
const written = await readFile(remotePath);
|
||||
expect(written.equals(payload)).toBe(true);
|
||||
|
||||
// Provider-backed sandbox runners cannot surface mid-flight progress for a
|
||||
// single stdin RPC, so we intentionally use several large append commands.
|
||||
expect(calls.length).toBeGreaterThan(2);
|
||||
expect(progress.length).toBeGreaterThan(2);
|
||||
for (let i = 1; i < progress.length; i++) {
|
||||
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
|
||||
}
|
||||
expect(progress.at(-1)).toEqual({ done: payload.length, total: payload.length });
|
||||
});
|
||||
|
||||
it("streams a download through onLog and reports monotonic byte progress to the total", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-read-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const remotePath = path.join(rootDir, "download.bin");
|
||||
|
||||
const payload = Buffer.alloc(512 * 1024);
|
||||
for (let i = 0; i < payload.length; i++) payload[i] = (i * 7) % 256;
|
||||
await writeFile(remotePath, payload);
|
||||
|
||||
const { runner } = makeSpawnRunner();
|
||||
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
||||
|
||||
const progress: Array<{ done: number; total: number | null }> = [];
|
||||
const bytes = await client.readFile(remotePath, {
|
||||
onProgress: (done, total) => {
|
||||
progress.push({ done, total });
|
||||
},
|
||||
});
|
||||
|
||||
expect(Buffer.from(bytes as ArrayBuffer).equals(payload)).toBe(true);
|
||||
|
||||
// The runner replays stdout in several chunks, so progress arrives in steps.
|
||||
expect(progress.length).toBeGreaterThan(1);
|
||||
for (let i = 1; i < progress.length; i++) {
|
||||
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
|
||||
}
|
||||
expect(progress.every((entry) => entry.total === payload.length)).toBe(true);
|
||||
expect(progress.at(-1)?.done).toBe(payload.length);
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user