feat(adapter-utils): add observable sandbox sync progress (#8395)
## Thinking Path > - Paperclip is the control plane for running AI-agent companies, so long-running remote work needs to stay observable to human operators. > - Cloud / sandbox agents are an active roadmap area, and their workspace sync path is part of the runtime substrate every remote coding run depends on. > - In the sandbox and SSH execution-target flows, Paperclip logged that sync had started, then often went silent for the full transfer window. > - That made large remote syncs feel stalled and also hid a real performance problem in the command-managed sandbox upload path. > - The first part of this pull request threads a throttled progress-reporting surface through the adapter execution-target stack so sync and restore work can emit meaningful updates. > - The second part fixes the command-managed sandbox transport itself: it removes the old serial 32KB append bottleneck, but also falls back away from the single-stream path when a provider-backed sandbox runner cannot surface mid-flight stdin progress. > - The result is that sandbox and SSH transfers are both faster and more observable, including the live Daytona-style sandbox case that previously only emitted `0%` and `100%`. ## Linked Issues or Issue Description No public GitHub issue exists for this bug, so it is described inline below following the bug report template. ### What happened - Remote sandbox and SSH workspace syncs could spend a long time transferring data while only logging a start line (`Syncing workspace and runtime assets to sandbox environment`) and, at best, a terminal line. - In the command-managed sandbox path, the original upload implementation also paid a large performance cost by appending base64 data in many small sequential remote writes (thousands of serial 32KB round-trips on a large workspace). - After the initial transport rewrite, live provider-backed sandbox runs still only emitted `0%` and `100%` because the single-stream stdin RPC buffered progress until completion. ### Expected behavior - Long-running sandbox and SSH syncs should periodically report how much of the transfer is complete (a percentage and/or MB transferred) so an operator can tell the run is healthy and making progress rather than stuck. - The main sandbox upload path should not be artificially slow. - A transfer that fails partway should leave an explicit failure marker in the log rather than a dangling intermediate percentage. ### Steps to reproduce 1. Run an agent against a sandbox (command-managed) or SSH (remote-managed) execution target with a non-trivial workspace. 2. Watch the run log during the workspace/runtime asset sync phase. 3. Observe that the log shows the sync start line and then stays silent for the full transfer (live provider-backed sandbox runs only show `0%` then `100%`). ### Paperclip version or commit - Branch `PAPA-825-provide-status-updates-when-syncing-sandboxes` off `master`. ### Deployment mode - Self-hosted / local instance using sandbox (command-managed) and SSH (remote-managed) execution targets, including provider-backed sandbox runners. ## What Changed - Added shared throttled runtime progress reporting and threaded `onProgress` through the adapter execution-target surface and adapter `execute.ts` entrypoints. - Added sync and restore progress reporting for the command-managed sandbox path and the SSH/remote-managed path, including git import/export progress where totals are known. - Reworked command-managed sandbox transfer behavior so uploads use the faster single-stream path when appropriate, but fall back to chunked progress-emitting writes when the runner cannot expose mid-stream stdin progress. - Marked provider-backed environment sandbox runners as not supporting single-stream stdin progress so live sandbox runs emit meaningful intermediate updates instead of only `0%` and `100%`. - Emit an explicit terminal failure marker (`failed at NN% (x/y MB)`) when an SSH/tar transfer rejects, so a failed sync no longer leaves a dangling intermediate percentage in the log. - Run the SSH sync/restore size estimate (local directory walk / remote `du` probe) concurrently with the transfer instead of awaiting it before opening the pipe, so progress instrumentation no longer adds startup latency proportional to workspace file count. - Added and extended focused regression coverage for runtime progress throttling and the new failure marker, command-managed sandbox transfers, sandbox orchestration, SSH transfer progress, and environment execution-target wiring. ## Verification - `pnpm exec vitest run packages/adapter-utils/src/runtime-progress.test.ts packages/adapter-utils/src/ssh-fixture.test.ts packages/adapter-utils/src/command-managed-runtime.test.ts packages/adapter-utils/src/sandbox-managed-runtime.test.ts` - `pnpm exec vitest run packages/adapter-utils/src/command-managed-runtime.test.ts server/src/__tests__/environment-execution-target.test.ts` - `npx tsc --noEmit` for `packages/adapter-utils` ## Risks - The provider-backed sandbox fallback now prefers chunked command-managed writes when progress hooks are active, so small-to-medium uploads may trade some raw throughput for observable intermediate progress on runtimes that cannot surface true mid-stream stdin progress. - Progress percentages on tar-based transfers still depend on estimates in some cases, so operators may briefly see MB-only lines before the estimate resolves, then near-final clamping before the terminal `100%` line. - This PR changes shared execution-target behavior used by multiple adapters, so regressions would most likely appear in remote runtime setup/teardown flows rather than in a single adapter. ## Model Used - Initial implementation: OpenAI GPT-5.4 via Codex local agent (`codex_local`), high reasoning mode. - Observability follow-ups (failure marker, concurrent size estimate, added tests): Claude Opus 4.8 via Claude Code (`claude_local`). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
@@ -11,6 +11,7 @@ import {
|
||||
readSshEnvLabFixtureStatus,
|
||||
restoreWorkspaceFromSshExecution,
|
||||
runSshCommand,
|
||||
syncDirectoryFromSsh,
|
||||
syncDirectoryToSsh,
|
||||
startSshEnvLabFixture,
|
||||
stopSshEnvLabFixture,
|
||||
@@ -54,6 +55,31 @@ async function startSshEnvLabFixtureOrSkip(statePath: string, label: string) {
|
||||
}
|
||||
}
|
||||
|
||||
interface ParsedProgressLine {
|
||||
raw: string;
|
||||
percent: number | null;
|
||||
doneMb: number | null;
|
||||
totalMb: number | null;
|
||||
}
|
||||
|
||||
function parseProgressLine(line: string): ParsedProgressLine {
|
||||
const trimmed = line.trimEnd();
|
||||
const percentMatch = trimmed.match(/:\s*(\d+)%\s*\(([\d.]+)\/([\d.]+) MB\)$/);
|
||||
if (percentMatch) {
|
||||
return {
|
||||
raw: trimmed,
|
||||
percent: Number.parseInt(percentMatch[1]!, 10),
|
||||
doneMb: Number.parseFloat(percentMatch[2]!),
|
||||
totalMb: Number.parseFloat(percentMatch[3]!),
|
||||
};
|
||||
}
|
||||
const mbMatch = trimmed.match(/:\s*([\d.]+) MB$/);
|
||||
if (mbMatch) {
|
||||
return { raw: trimmed, percent: null, doneMb: Number.parseFloat(mbMatch[1]!), totalMb: null };
|
||||
}
|
||||
return { raw: trimmed, percent: null, doneMb: null, totalMb: null };
|
||||
}
|
||||
|
||||
describe("ssh env-lab fixture", () => {
|
||||
const cleanupDirs: string[] = [];
|
||||
|
||||
@@ -199,6 +225,142 @@ describe("ssh env-lab fixture", () => {
|
||||
expect(result.stdout).not.toContain("appledouble-present");
|
||||
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
||||
|
||||
it("reports throttled upload progress with a clamped percent and terminal 100% line", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const statePath = path.join(rootDir, "state.json");
|
||||
const localDir = path.join(rootDir, "local-overlay");
|
||||
|
||||
await mkdir(localDir, { recursive: true });
|
||||
// Multiple files large enough that tar emits several pipe chunks, so the
|
||||
// byte counter crosses several step boundaries before the stream closes.
|
||||
for (let index = 0; index < 4; index += 1) {
|
||||
await writeFile(path.join(localDir, `blob-${index}.bin`), Buffer.alloc(256 * 1024, index + 1));
|
||||
}
|
||||
|
||||
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH upload progress test");
|
||||
if (!started) return;
|
||||
const config = await buildSshEnvLabFixtureConfig(started);
|
||||
const remoteDir = path.posix.join(started.workspaceDir, "overlay-progress");
|
||||
|
||||
const lines: ParsedProgressLine[] = [];
|
||||
await syncDirectoryToSsh({
|
||||
spec: { ...config, remoteCwd: started.workspaceDir },
|
||||
localDir,
|
||||
remoteDir,
|
||||
onProgress: (line) => {
|
||||
lines.push(parseProgressLine(line));
|
||||
},
|
||||
progressLabel: "workspace",
|
||||
});
|
||||
|
||||
expect(lines.length).toBeGreaterThan(0);
|
||||
for (const line of lines) {
|
||||
expect(line.raw).toContain("Syncing workspace to ssh");
|
||||
}
|
||||
// Monotonically increasing byte counts.
|
||||
const doneSeries = lines.map((line) => line.doneMb ?? 0);
|
||||
for (let index = 1; index < doneSeries.length; index += 1) {
|
||||
expect(doneSeries[index]!).toBeGreaterThanOrEqual(doneSeries[index - 1]!);
|
||||
}
|
||||
// Percent clamped to <= 99% on every line emitted before the stream closed.
|
||||
for (const line of lines.slice(0, -1)) {
|
||||
if (line.percent != null) expect(line.percent).toBeLessThanOrEqual(99);
|
||||
}
|
||||
// Terminal completion line is 100% with matching done/total.
|
||||
const last = lines.at(-1)!;
|
||||
expect(last.percent).toBe(100);
|
||||
expect(last.doneMb).toBe(last.totalMb);
|
||||
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
||||
|
||||
it("reports restore progress with a terminal completion line", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const statePath = path.join(rootDir, "state.json");
|
||||
const localDir = path.join(rootDir, "local-overlay");
|
||||
const restoreDir = path.join(rootDir, "restore-target");
|
||||
|
||||
await mkdir(localDir, { recursive: true });
|
||||
await mkdir(restoreDir, { recursive: true });
|
||||
for (let index = 0; index < 4; index += 1) {
|
||||
await writeFile(path.join(localDir, `blob-${index}.bin`), Buffer.alloc(256 * 1024, index + 1));
|
||||
}
|
||||
|
||||
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH restore progress test");
|
||||
if (!started) return;
|
||||
const config = await buildSshEnvLabFixtureConfig(started);
|
||||
const spec = { ...config, remoteCwd: started.workspaceDir } as const;
|
||||
const remoteDir = path.posix.join(started.workspaceDir, "restore-source");
|
||||
|
||||
await syncDirectoryToSsh({ spec, localDir, remoteDir });
|
||||
|
||||
const lines: ParsedProgressLine[] = [];
|
||||
await syncDirectoryFromSsh({
|
||||
spec,
|
||||
remoteDir,
|
||||
localDir: restoreDir,
|
||||
onProgress: (line) => {
|
||||
lines.push(parseProgressLine(line));
|
||||
},
|
||||
progressLabel: "workspace",
|
||||
});
|
||||
|
||||
expect(lines.length).toBeGreaterThan(0);
|
||||
for (const line of lines) {
|
||||
expect(line.raw).toContain("Restoring workspace from ssh");
|
||||
}
|
||||
// Terminal completion line: either an exact 100% (probe succeeded) or a
|
||||
// final MB-received line (probe unavailable). Either is a valid terminal.
|
||||
const last = lines.at(-1)!;
|
||||
expect(last.percent === 100 || (last.percent === null && last.doneMb !== null)).toBe(true);
|
||||
// The restored files round-tripped through the byte-counting transport.
|
||||
await expect(readFile(path.join(restoreDir, "blob-0.bin"))).resolves.toEqual(
|
||||
Buffer.alloc(256 * 1024, 1),
|
||||
);
|
||||
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
||||
|
||||
it("reports exact git-history import percentage from the known bundle size", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const statePath = path.join(rootDir, "state.json");
|
||||
const localRepo = path.join(rootDir, "local-workspace");
|
||||
|
||||
await mkdir(localRepo, { recursive: true });
|
||||
await git(localRepo, ["init"]);
|
||||
await git(localRepo, ["checkout", "-b", "main"]);
|
||||
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
|
||||
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
|
||||
await writeFile(path.join(localRepo, "tracked.bin"), Buffer.alloc(256 * 1024, 7));
|
||||
await git(localRepo, ["add", "tracked.bin"]);
|
||||
await git(localRepo, ["commit", "-m", "initial"]);
|
||||
|
||||
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH git import progress test");
|
||||
if (!started) return;
|
||||
const config = await buildSshEnvLabFixtureConfig(started);
|
||||
const spec = { ...config, remoteCwd: started.workspaceDir } as const;
|
||||
|
||||
const lines: ParsedProgressLine[] = [];
|
||||
await prepareWorkspaceForSshExecution({
|
||||
spec,
|
||||
localDir: localRepo,
|
||||
remoteDir: started.workspaceDir,
|
||||
onProgress: (line) => {
|
||||
lines.push(parseProgressLine(line));
|
||||
},
|
||||
});
|
||||
|
||||
const importLines = lines.filter((line) => line.raw.includes("Importing git history to ssh"));
|
||||
expect(importLines.length).toBeGreaterThan(0);
|
||||
// Known bundle size -> exact percentage with no "workspace" label.
|
||||
for (const line of importLines) {
|
||||
expect(line.raw).not.toContain("workspace");
|
||||
expect(line.percent).not.toBeNull();
|
||||
}
|
||||
const lastImport = importLines.at(-1)!;
|
||||
expect(lastImport.percent).toBe(100);
|
||||
expect(lastImport.doneMb).toBe(lastImport.totalMb);
|
||||
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
||||
|
||||
it("can dereference local symlinks while syncing to the remote fixture", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
||||
Reference in New Issue
Block a user