07e98d2b2c
## Thinking Path > - Paperclip is the control plane for running AI-agent companies, so long-running remote work needs to stay observable to human operators. > - Cloud / sandbox agents are an active roadmap area, and their workspace sync path is part of the runtime substrate every remote coding run depends on. > - In the sandbox and SSH execution-target flows, Paperclip logged that sync had started, then often went silent for the full transfer window. > - That made large remote syncs feel stalled and also hid a real performance problem in the command-managed sandbox upload path. > - The first part of this pull request threads a throttled progress-reporting surface through the adapter execution-target stack so sync and restore work can emit meaningful updates. > - The second part fixes the command-managed sandbox transport itself: it removes the old serial 32KB append bottleneck, but also falls back away from the single-stream path when a provider-backed sandbox runner cannot surface mid-flight stdin progress. > - The result is that sandbox and SSH transfers are both faster and more observable, including the live Daytona-style sandbox case that previously only emitted `0%` and `100%`. ## Linked Issues or Issue Description No public GitHub issue exists for this bug, so it is described inline below following the bug report template. ### What happened - Remote sandbox and SSH workspace syncs could spend a long time transferring data while only logging a start line (`Syncing workspace and runtime assets to sandbox environment`) and, at best, a terminal line. - In the command-managed sandbox path, the original upload implementation also paid a large performance cost by appending base64 data in many small sequential remote writes (thousands of serial 32KB round-trips on a large workspace). - After the initial transport rewrite, live provider-backed sandbox runs still only emitted `0%` and `100%` because the single-stream stdin RPC buffered progress until completion. ### Expected behavior - Long-running sandbox and SSH syncs should periodically report how much of the transfer is complete (a percentage and/or MB transferred) so an operator can tell the run is healthy and making progress rather than stuck. - The main sandbox upload path should not be artificially slow. - A transfer that fails partway should leave an explicit failure marker in the log rather than a dangling intermediate percentage. ### Steps to reproduce 1. Run an agent against a sandbox (command-managed) or SSH (remote-managed) execution target with a non-trivial workspace. 2. Watch the run log during the workspace/runtime asset sync phase. 3. Observe that the log shows the sync start line and then stays silent for the full transfer (live provider-backed sandbox runs only show `0%` then `100%`). ### Paperclip version or commit - Branch `PAPA-825-provide-status-updates-when-syncing-sandboxes` off `master`. ### Deployment mode - Self-hosted / local instance using sandbox (command-managed) and SSH (remote-managed) execution targets, including provider-backed sandbox runners. ## What Changed - Added shared throttled runtime progress reporting and threaded `onProgress` through the adapter execution-target surface and adapter `execute.ts` entrypoints. - Added sync and restore progress reporting for the command-managed sandbox path and the SSH/remote-managed path, including git import/export progress where totals are known. - Reworked command-managed sandbox transfer behavior so uploads use the faster single-stream path when appropriate, but fall back to chunked progress-emitting writes when the runner cannot expose mid-stream stdin progress. - Marked provider-backed environment sandbox runners as not supporting single-stream stdin progress so live sandbox runs emit meaningful intermediate updates instead of only `0%` and `100%`. - Emit an explicit terminal failure marker (`failed at NN% (x/y MB)`) when an SSH/tar transfer rejects, so a failed sync no longer leaves a dangling intermediate percentage in the log. - Run the SSH sync/restore size estimate (local directory walk / remote `du` probe) concurrently with the transfer instead of awaiting it before opening the pipe, so progress instrumentation no longer adds startup latency proportional to workspace file count. - Added and extended focused regression coverage for runtime progress throttling and the new failure marker, command-managed sandbox transfers, sandbox orchestration, SSH transfer progress, and environment execution-target wiring. ## Verification - `pnpm exec vitest run packages/adapter-utils/src/runtime-progress.test.ts packages/adapter-utils/src/ssh-fixture.test.ts packages/adapter-utils/src/command-managed-runtime.test.ts packages/adapter-utils/src/sandbox-managed-runtime.test.ts` - `pnpm exec vitest run packages/adapter-utils/src/command-managed-runtime.test.ts server/src/__tests__/environment-execution-target.test.ts` - `npx tsc --noEmit` for `packages/adapter-utils` ## Risks - The provider-backed sandbox fallback now prefers chunked command-managed writes when progress hooks are active, so small-to-medium uploads may trade some raw throughput for observable intermediate progress on runtimes that cannot surface true mid-stream stdin progress. - Progress percentages on tar-based transfers still depend on estimates in some cases, so operators may briefly see MB-only lines before the estimate resolves, then near-final clamping before the terminal `100%` line. - This PR changes shared execution-target behavior used by multiple adapters, so regressions would most likely appear in remote runtime setup/teardown flows rather than in a single adapter. ## Model Used - Initial implementation: OpenAI GPT-5.4 via Codex local agent (`codex_local`), high reasoning mode. - Observability follow-ups (failure marker, concurrent size estimate, added tests): Claude Opus 4.8 via Claude Code (`claude_local`). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
739 lines
30 KiB
TypeScript
739 lines
30 KiB
TypeScript
import { execFile } from "node:child_process";
|
|
import { mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { afterEach, describe, expect, it } from "vitest";
|
|
import {
|
|
buildSshSpawnTarget,
|
|
buildSshEnvLabFixtureConfig,
|
|
getSshEnvLabSupport,
|
|
prepareWorkspaceForSshExecution,
|
|
readSshEnvLabFixtureStatus,
|
|
restoreWorkspaceFromSshExecution,
|
|
runSshCommand,
|
|
syncDirectoryFromSsh,
|
|
syncDirectoryToSsh,
|
|
startSshEnvLabFixture,
|
|
stopSshEnvLabFixture,
|
|
} from "./ssh.js";
|
|
import { prepareRemoteManagedRuntime } from "./remote-managed-runtime.js";
|
|
|
|
const SSH_FIXTURE_TEST_TIMEOUT_MS = 30_000;
|
|
let sshEnvLabUnsupportedReason: string | null = null;
|
|
|
|
async function git(cwd: string, args: string[]): Promise<string> {
|
|
return await new Promise((resolve, reject) => {
|
|
execFile("git", ["-C", cwd, ...args], (error, stdout, stderr) => {
|
|
if (error) {
|
|
reject(new Error((stderr || stdout || error.message).trim()));
|
|
return;
|
|
}
|
|
resolve(stdout.trim());
|
|
});
|
|
});
|
|
}
|
|
|
|
async function startSshEnvLabFixtureOrSkip(statePath: string, label: string) {
|
|
if (sshEnvLabUnsupportedReason) {
|
|
console.warn(`Skipping ${label}: ${sshEnvLabUnsupportedReason}`);
|
|
return null;
|
|
}
|
|
|
|
const support = await getSshEnvLabSupport();
|
|
if (!support.supported) {
|
|
sshEnvLabUnsupportedReason = support.reason ?? "unsupported environment";
|
|
console.warn(`Skipping ${label}: ${sshEnvLabUnsupportedReason}`);
|
|
return null;
|
|
}
|
|
|
|
try {
|
|
return await startSshEnvLabFixture({ statePath });
|
|
} catch (error) {
|
|
sshEnvLabUnsupportedReason = error instanceof Error ? error.message : String(error);
|
|
console.warn(`Skipping ${label}: ${sshEnvLabUnsupportedReason}`);
|
|
return null;
|
|
}
|
|
}
|
|
|
|
interface ParsedProgressLine {
|
|
raw: string;
|
|
percent: number | null;
|
|
doneMb: number | null;
|
|
totalMb: number | null;
|
|
}
|
|
|
|
function parseProgressLine(line: string): ParsedProgressLine {
|
|
const trimmed = line.trimEnd();
|
|
const percentMatch = trimmed.match(/:\s*(\d+)%\s*\(([\d.]+)\/([\d.]+) MB\)$/);
|
|
if (percentMatch) {
|
|
return {
|
|
raw: trimmed,
|
|
percent: Number.parseInt(percentMatch[1]!, 10),
|
|
doneMb: Number.parseFloat(percentMatch[2]!),
|
|
totalMb: Number.parseFloat(percentMatch[3]!),
|
|
};
|
|
}
|
|
const mbMatch = trimmed.match(/:\s*([\d.]+) MB$/);
|
|
if (mbMatch) {
|
|
return { raw: trimmed, percent: null, doneMb: Number.parseFloat(mbMatch[1]!), totalMb: null };
|
|
}
|
|
return { raw: trimmed, percent: null, doneMb: null, totalMb: null };
|
|
}
|
|
|
|
describe("ssh env-lab fixture", () => {
|
|
const cleanupDirs: string[] = [];
|
|
|
|
afterEach(async () => {
|
|
while (cleanupDirs.length > 0) {
|
|
const dir = cleanupDirs.pop();
|
|
if (!dir) continue;
|
|
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
|
|
}
|
|
});
|
|
|
|
it("starts an isolated sshd fixture and executes commands through it", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH env-lab fixture test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const quotedWorkspace = JSON.stringify(started.workspaceDir);
|
|
const result = await runSshCommand(
|
|
config,
|
|
`cd ${quotedWorkspace} && pwd`,
|
|
);
|
|
|
|
expect(result.stdout.trim()).toBe(started.workspaceDir);
|
|
const status = await readSshEnvLabFixtureStatus(statePath);
|
|
expect(status.running).toBe(true);
|
|
|
|
await stopSshEnvLabFixture(statePath);
|
|
|
|
const stopped = await readSshEnvLabFixtureStatus(statePath);
|
|
expect(stopped.running).toBe(false);
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("forwards stdin to remote SSH commands", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH stdin forwarding test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const remotePath = path.posix.join(started.workspaceDir, "stdin-forwarded.txt");
|
|
|
|
await runSshCommand(
|
|
config,
|
|
`cat > ${JSON.stringify(remotePath)}`,
|
|
{
|
|
stdin: "hello over ssh stdin\n",
|
|
timeoutMs: 30_000,
|
|
maxBuffer: 256 * 1024,
|
|
},
|
|
);
|
|
|
|
const result = await runSshCommand(
|
|
config,
|
|
`cat ${JSON.stringify(remotePath)}`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
|
|
expect(result.stdout).toBe("hello over ssh stdin\n");
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("does not treat an unrelated reused pid as the running fixture", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH env-lab fixture test");
|
|
if (!started) return;
|
|
await stopSshEnvLabFixture(statePath);
|
|
await mkdir(path.dirname(statePath), { recursive: true });
|
|
|
|
await writeFile(
|
|
statePath,
|
|
JSON.stringify({ ...started, pid: process.pid }, null, 2),
|
|
{ mode: 0o600 },
|
|
);
|
|
|
|
const staleStatus = await readSshEnvLabFixtureStatus(statePath);
|
|
expect(staleStatus.running).toBe(false);
|
|
|
|
const restarted = await startSshEnvLabFixtureOrSkip(statePath, "SSH env-lab fixture restart test");
|
|
if (!restarted) return;
|
|
expect(restarted.pid).not.toBe(process.pid);
|
|
|
|
await stopSshEnvLabFixture(statePath);
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("rejects invalid environment variable keys when constructing SSH spawn targets", async () => {
|
|
await expect(
|
|
buildSshSpawnTarget({
|
|
spec: {
|
|
host: "ssh.example.test",
|
|
port: 22,
|
|
username: "ssh-user",
|
|
remoteCwd: "/srv/paperclip/workspace",
|
|
remoteWorkspacePath: "/srv/paperclip/workspace",
|
|
privateKey: null,
|
|
knownHosts: null,
|
|
strictHostKeyChecking: true,
|
|
},
|
|
command: "env",
|
|
args: [],
|
|
env: {
|
|
"BAD KEY": "value",
|
|
},
|
|
}),
|
|
).rejects.toThrow("Invalid SSH environment variable key: BAD KEY");
|
|
});
|
|
|
|
it("syncs a local directory into the remote fixture workspace", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localDir = path.join(rootDir, "local-overlay");
|
|
|
|
await mkdir(localDir, { recursive: true });
|
|
await writeFile(path.join(localDir, "message.txt"), "hello from paperclip\n", "utf8");
|
|
await writeFile(path.join(localDir, "._message.txt"), "should never sync\n", "utf8");
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH env-lab fixture test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const remoteDir = path.posix.join(started.workspaceDir, "overlay");
|
|
|
|
await syncDirectoryToSsh({
|
|
spec: {
|
|
...config,
|
|
remoteCwd: started.workspaceDir,
|
|
},
|
|
localDir,
|
|
remoteDir,
|
|
});
|
|
|
|
const result = await runSshCommand(
|
|
config,
|
|
`cat ${JSON.stringify(path.posix.join(remoteDir, "message.txt"))} && if [ -e ${JSON.stringify(path.posix.join(remoteDir, "._message.txt"))} ]; then echo appledouble-present; fi`,
|
|
);
|
|
|
|
expect(result.stdout).toContain("hello from paperclip");
|
|
expect(result.stdout).not.toContain("appledouble-present");
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("reports throttled upload progress with a clamped percent and terminal 100% line", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localDir = path.join(rootDir, "local-overlay");
|
|
|
|
await mkdir(localDir, { recursive: true });
|
|
// Multiple files large enough that tar emits several pipe chunks, so the
|
|
// byte counter crosses several step boundaries before the stream closes.
|
|
for (let index = 0; index < 4; index += 1) {
|
|
await writeFile(path.join(localDir, `blob-${index}.bin`), Buffer.alloc(256 * 1024, index + 1));
|
|
}
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH upload progress test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const remoteDir = path.posix.join(started.workspaceDir, "overlay-progress");
|
|
|
|
const lines: ParsedProgressLine[] = [];
|
|
await syncDirectoryToSsh({
|
|
spec: { ...config, remoteCwd: started.workspaceDir },
|
|
localDir,
|
|
remoteDir,
|
|
onProgress: (line) => {
|
|
lines.push(parseProgressLine(line));
|
|
},
|
|
progressLabel: "workspace",
|
|
});
|
|
|
|
expect(lines.length).toBeGreaterThan(0);
|
|
for (const line of lines) {
|
|
expect(line.raw).toContain("Syncing workspace to ssh");
|
|
}
|
|
// Monotonically increasing byte counts.
|
|
const doneSeries = lines.map((line) => line.doneMb ?? 0);
|
|
for (let index = 1; index < doneSeries.length; index += 1) {
|
|
expect(doneSeries[index]!).toBeGreaterThanOrEqual(doneSeries[index - 1]!);
|
|
}
|
|
// Percent clamped to <= 99% on every line emitted before the stream closed.
|
|
for (const line of lines.slice(0, -1)) {
|
|
if (line.percent != null) expect(line.percent).toBeLessThanOrEqual(99);
|
|
}
|
|
// Terminal completion line is 100% with matching done/total.
|
|
const last = lines.at(-1)!;
|
|
expect(last.percent).toBe(100);
|
|
expect(last.doneMb).toBe(last.totalMb);
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("reports restore progress with a terminal completion line", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localDir = path.join(rootDir, "local-overlay");
|
|
const restoreDir = path.join(rootDir, "restore-target");
|
|
|
|
await mkdir(localDir, { recursive: true });
|
|
await mkdir(restoreDir, { recursive: true });
|
|
for (let index = 0; index < 4; index += 1) {
|
|
await writeFile(path.join(localDir, `blob-${index}.bin`), Buffer.alloc(256 * 1024, index + 1));
|
|
}
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH restore progress test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const spec = { ...config, remoteCwd: started.workspaceDir } as const;
|
|
const remoteDir = path.posix.join(started.workspaceDir, "restore-source");
|
|
|
|
await syncDirectoryToSsh({ spec, localDir, remoteDir });
|
|
|
|
const lines: ParsedProgressLine[] = [];
|
|
await syncDirectoryFromSsh({
|
|
spec,
|
|
remoteDir,
|
|
localDir: restoreDir,
|
|
onProgress: (line) => {
|
|
lines.push(parseProgressLine(line));
|
|
},
|
|
progressLabel: "workspace",
|
|
});
|
|
|
|
expect(lines.length).toBeGreaterThan(0);
|
|
for (const line of lines) {
|
|
expect(line.raw).toContain("Restoring workspace from ssh");
|
|
}
|
|
// Terminal completion line: either an exact 100% (probe succeeded) or a
|
|
// final MB-received line (probe unavailable). Either is a valid terminal.
|
|
const last = lines.at(-1)!;
|
|
expect(last.percent === 100 || (last.percent === null && last.doneMb !== null)).toBe(true);
|
|
// The restored files round-tripped through the byte-counting transport.
|
|
await expect(readFile(path.join(restoreDir, "blob-0.bin"))).resolves.toEqual(
|
|
Buffer.alloc(256 * 1024, 1),
|
|
);
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("reports exact git-history import percentage from the known bundle size", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localRepo = path.join(rootDir, "local-workspace");
|
|
|
|
await mkdir(localRepo, { recursive: true });
|
|
await git(localRepo, ["init"]);
|
|
await git(localRepo, ["checkout", "-b", "main"]);
|
|
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
|
|
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
|
|
await writeFile(path.join(localRepo, "tracked.bin"), Buffer.alloc(256 * 1024, 7));
|
|
await git(localRepo, ["add", "tracked.bin"]);
|
|
await git(localRepo, ["commit", "-m", "initial"]);
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH git import progress test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const spec = { ...config, remoteCwd: started.workspaceDir } as const;
|
|
|
|
const lines: ParsedProgressLine[] = [];
|
|
await prepareWorkspaceForSshExecution({
|
|
spec,
|
|
localDir: localRepo,
|
|
remoteDir: started.workspaceDir,
|
|
onProgress: (line) => {
|
|
lines.push(parseProgressLine(line));
|
|
},
|
|
});
|
|
|
|
const importLines = lines.filter((line) => line.raw.includes("Importing git history to ssh"));
|
|
expect(importLines.length).toBeGreaterThan(0);
|
|
// Known bundle size -> exact percentage with no "workspace" label.
|
|
for (const line of importLines) {
|
|
expect(line.raw).not.toContain("workspace");
|
|
expect(line.percent).not.toBeNull();
|
|
}
|
|
const lastImport = importLines.at(-1)!;
|
|
expect(lastImport.percent).toBe(100);
|
|
expect(lastImport.doneMb).toBe(lastImport.totalMb);
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("can dereference local symlinks while syncing to the remote fixture", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const sourceDir = path.join(rootDir, "source");
|
|
const localDir = path.join(rootDir, "local-overlay");
|
|
|
|
await mkdir(sourceDir, { recursive: true });
|
|
await mkdir(localDir, { recursive: true });
|
|
await writeFile(path.join(sourceDir, "auth.json"), "{\"token\":\"secret\"}\n", "utf8");
|
|
await symlink(path.join(sourceDir, "auth.json"), path.join(localDir, "auth.json"));
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH symlink sync test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const remoteDir = path.posix.join(started.workspaceDir, "overlay-follow-links");
|
|
|
|
await syncDirectoryToSsh({
|
|
spec: {
|
|
...config,
|
|
remoteCwd: started.workspaceDir,
|
|
},
|
|
localDir,
|
|
remoteDir,
|
|
followSymlinks: true,
|
|
});
|
|
|
|
const result = await runSshCommand(
|
|
config,
|
|
`if [ -L ${JSON.stringify(path.posix.join(remoteDir, "auth.json"))} ]; then echo symlink; else echo regular; fi && cat ${JSON.stringify(path.posix.join(remoteDir, "auth.json"))}`,
|
|
);
|
|
|
|
expect(result.stdout).toContain("regular");
|
|
expect(result.stdout).toContain("{\"token\":\"secret\"}");
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("round-trips a git workspace through the SSH fixture", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localRepo = path.join(rootDir, "local-workspace");
|
|
|
|
await mkdir(localRepo, { recursive: true });
|
|
await git(localRepo, ["init"]);
|
|
await git(localRepo, ["checkout", "-b", "main"]);
|
|
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
|
|
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
|
|
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
|
|
await writeFile(path.join(localRepo, "._tracked.txt"), "should stay local only\n", "utf8");
|
|
await git(localRepo, ["add", "tracked.txt"]);
|
|
await git(localRepo, ["commit", "-m", "initial"]);
|
|
const originalHead = await git(localRepo, ["rev-parse", "HEAD"]);
|
|
await writeFile(path.join(localRepo, "tracked.txt"), "dirty local\n", "utf8");
|
|
await writeFile(path.join(localRepo, "untracked.txt"), "from local\n", "utf8");
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH workspace round-trip test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const spec = {
|
|
...config,
|
|
remoteCwd: started.workspaceDir,
|
|
} as const;
|
|
|
|
await prepareWorkspaceForSshExecution({
|
|
spec,
|
|
localDir: localRepo,
|
|
remoteDir: started.workspaceDir,
|
|
});
|
|
|
|
const remoteStatus = await runSshCommand(
|
|
config,
|
|
`cd ${JSON.stringify(started.workspaceDir)} && git status --short`,
|
|
);
|
|
expect(remoteStatus.stdout).toContain("M tracked.txt");
|
|
expect(remoteStatus.stdout).toContain("?? untracked.txt");
|
|
expect(remoteStatus.stdout).not.toContain("._tracked.txt");
|
|
|
|
await runSshCommand(
|
|
config,
|
|
`cd ${JSON.stringify(started.workspaceDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && git add tracked.txt untracked.txt && git commit -m "remote update" >/dev/null && printf "remote dirty\\n" > tracked.txt && printf "remote extra\\n" > remote-only.txt`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
|
|
await restoreWorkspaceFromSshExecution({
|
|
spec,
|
|
localDir: localRepo,
|
|
remoteDir: started.workspaceDir,
|
|
});
|
|
|
|
const restoredHead = await git(localRepo, ["rev-parse", "HEAD"]);
|
|
expect(restoredHead).not.toBe(originalHead);
|
|
expect(await git(localRepo, ["log", "-1", "--pretty=%s"])).toBe("remote update");
|
|
expect(await git(localRepo, ["status", "--short"])).toContain("M tracked.txt");
|
|
expect(await git(localRepo, ["status", "--short"])).not.toContain("._tracked.txt");
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("preserves both concurrent SSH restores in a shared git workspace", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localRepo = path.join(rootDir, "local-workspace");
|
|
|
|
await mkdir(localRepo, { recursive: true });
|
|
await git(localRepo, ["init"]);
|
|
await git(localRepo, ["checkout", "-b", "main"]);
|
|
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
|
|
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
|
|
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
|
|
await git(localRepo, ["add", "tracked.txt"]);
|
|
await git(localRepo, ["commit", "-m", "initial"]);
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "concurrent SSH restore test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const spec = {
|
|
...config,
|
|
remoteCwd: started.workspaceDir,
|
|
} as const;
|
|
|
|
const preparedA = await prepareRemoteManagedRuntime({
|
|
spec,
|
|
runId: "run-a",
|
|
adapterKey: "test-adapter",
|
|
workspaceLocalDir: localRepo,
|
|
});
|
|
const preparedB = await prepareRemoteManagedRuntime({
|
|
spec,
|
|
runId: "run-b",
|
|
adapterKey: "test-adapter",
|
|
workspaceLocalDir: localRepo,
|
|
});
|
|
|
|
expect(preparedA.workspaceRemoteDir).not.toBe(preparedB.workspaceRemoteDir);
|
|
|
|
await runSshCommand(
|
|
config,
|
|
`printf "from run a\\n" > ${JSON.stringify(path.posix.join(preparedA.workspaceRemoteDir, "run-a.txt"))}`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
await runSshCommand(
|
|
config,
|
|
`printf "from run b\\n" > ${JSON.stringify(path.posix.join(preparedB.workspaceRemoteDir, "run-b.txt"))}`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
|
|
await Promise.all([
|
|
preparedA.restoreWorkspace(),
|
|
preparedB.restoreWorkspace(),
|
|
]);
|
|
|
|
await expect(readFile(path.join(localRepo, "run-a.txt"), "utf8")).resolves.toBe("from run a\n");
|
|
await expect(readFile(path.join(localRepo, "run-b.txt"), "utf8")).resolves.toBe("from run b\n");
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("preserves nested per-run files across sequential SSH restores with stale baselines", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localRepo = path.join(rootDir, "local-workspace");
|
|
|
|
await mkdir(localRepo, { recursive: true });
|
|
await git(localRepo, ["init"]);
|
|
await git(localRepo, ["checkout", "-b", "main"]);
|
|
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
|
|
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
|
|
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
|
|
await git(localRepo, ["add", "tracked.txt"]);
|
|
await git(localRepo, ["commit", "-m", "initial"]);
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "sequential nested SSH restore test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const spec = {
|
|
...config,
|
|
remoteCwd: started.workspaceDir,
|
|
} as const;
|
|
|
|
const preparedA = await prepareRemoteManagedRuntime({
|
|
spec,
|
|
runId: "run-a",
|
|
adapterKey: "test-adapter",
|
|
workspaceLocalDir: localRepo,
|
|
});
|
|
const preparedB = await prepareRemoteManagedRuntime({
|
|
spec,
|
|
runId: "run-b",
|
|
adapterKey: "test-adapter",
|
|
workspaceLocalDir: localRepo,
|
|
});
|
|
|
|
await runSshCommand(
|
|
config,
|
|
`mkdir -p ${JSON.stringify(path.posix.join(preparedA.workspaceRemoteDir, "manual-qa/environment-matrix/ssh"))} && printf "from run a\\n" > ${JSON.stringify(path.posix.join(preparedA.workspaceRemoteDir, "manual-qa/environment-matrix/ssh/claude_local.md"))}`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
await runSshCommand(
|
|
config,
|
|
`mkdir -p ${JSON.stringify(path.posix.join(preparedB.workspaceRemoteDir, "manual-qa/environment-matrix/ssh"))} && printf "from run b\\n" > ${JSON.stringify(path.posix.join(preparedB.workspaceRemoteDir, "manual-qa/environment-matrix/ssh/codex_local.md"))}`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
|
|
await preparedA.restoreWorkspace();
|
|
await preparedB.restoreWorkspace();
|
|
|
|
await expect(readFile(path.join(localRepo, "manual-qa/environment-matrix/ssh/claude_local.md"), "utf8")).resolves
|
|
.toBe("from run a\n");
|
|
await expect(readFile(path.join(localRepo, "manual-qa/environment-matrix/ssh/codex_local.md"), "utf8")).resolves
|
|
.toBe("from run b\n");
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("round-trips remote git commits through the managed runtime restore path", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localRepo = path.join(rootDir, "local-workspace");
|
|
|
|
await mkdir(localRepo, { recursive: true });
|
|
await git(localRepo, ["init"]);
|
|
await git(localRepo, ["checkout", "-b", "main"]);
|
|
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
|
|
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
|
|
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
|
|
await git(localRepo, ["add", "tracked.txt"]);
|
|
await git(localRepo, ["commit", "-m", "initial"]);
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "managed-runtime SSH git round-trip test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const spec = {
|
|
...config,
|
|
remoteCwd: started.workspaceDir,
|
|
} as const;
|
|
|
|
const prepared = await prepareRemoteManagedRuntime({
|
|
spec,
|
|
runId: "run-commit",
|
|
adapterKey: "test-adapter",
|
|
workspaceLocalDir: localRepo,
|
|
});
|
|
|
|
await runSshCommand(
|
|
config,
|
|
`cd ${JSON.stringify(prepared.workspaceRemoteDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && printf "committed\\n" > tracked.txt && git add tracked.txt && git commit -m "remote update" >/dev/null && printf "dirty remote\\n" > tracked.txt`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
|
|
await prepared.restoreWorkspace();
|
|
|
|
expect(await git(localRepo, ["log", "-1", "--pretty=%s"])).toBe("remote update");
|
|
await expect(readFile(path.join(localRepo, "tracked.txt"), "utf8")).resolves.toBe("dirty remote\n");
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("propagates remote commits to the local worktree with no git remote configured (no-remote-git contract)", async () => {
|
|
// Locks in the architectural contract documented in
|
|
// packages/adapter-utils/README.md and packages/adapters/AUTHORING.md:
|
|
// the local execution-workspace cwd is the only persistence boundary
|
|
// across runs. No adapter may depend on a git remote for cross-run state.
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localRepo = path.join(rootDir, "local-workspace");
|
|
|
|
await mkdir(localRepo, { recursive: true });
|
|
await git(localRepo, ["init"]);
|
|
await git(localRepo, ["checkout", "-b", "main"]);
|
|
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
|
|
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
|
|
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
|
|
await git(localRepo, ["add", "tracked.txt"]);
|
|
await git(localRepo, ["commit", "-m", "initial"]);
|
|
|
|
// Assert there is no git remote configured before we begin, and verify
|
|
// that no point in the round-trip introduces one. `git remote` returns an
|
|
// empty string when no remotes exist (and exit code 0).
|
|
expect(await git(localRepo, ["remote"])).toBe("");
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(
|
|
statePath,
|
|
"no-remote-git contract test",
|
|
);
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const spec = {
|
|
...config,
|
|
remoteCwd: started.workspaceDir,
|
|
} as const;
|
|
|
|
const prepared = await prepareRemoteManagedRuntime({
|
|
spec,
|
|
runId: "run-no-remote",
|
|
adapterKey: "test-adapter",
|
|
workspaceLocalDir: localRepo,
|
|
});
|
|
|
|
// Remote commit lands a deliverable that must show up locally via
|
|
// sync-back alone — no `git push`, no fetch from any origin.
|
|
await runSshCommand(
|
|
config,
|
|
`cd ${JSON.stringify(prepared.workspaceRemoteDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && printf "deliverable\\n" > tracked.txt && git add tracked.txt && git commit -m "remote-only commit" >/dev/null`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
|
|
await prepared.restoreWorkspace();
|
|
|
|
expect(await git(localRepo, ["log", "-1", "--pretty=%s"])).toBe(
|
|
"remote-only commit",
|
|
);
|
|
expect(await readFile(path.join(localRepo, "tracked.txt"), "utf8")).toBe(
|
|
"deliverable\n",
|
|
);
|
|
// Final assertion: still no git remote — restore did not silently add one.
|
|
expect(await git(localRepo, ["remote"])).toBe("");
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
|
|
it("merges concurrent remote commits through the managed runtime restore path", async () => {
|
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
|
|
cleanupDirs.push(rootDir);
|
|
const statePath = path.join(rootDir, "state.json");
|
|
const localRepo = path.join(rootDir, "local-workspace");
|
|
|
|
await mkdir(localRepo, { recursive: true });
|
|
await git(localRepo, ["init"]);
|
|
await git(localRepo, ["checkout", "-b", "main"]);
|
|
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
|
|
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
|
|
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
|
|
await git(localRepo, ["add", "tracked.txt"]);
|
|
await git(localRepo, ["commit", "-m", "initial"]);
|
|
|
|
const started = await startSshEnvLabFixtureOrSkip(statePath, "concurrent managed-runtime SSH git merge test");
|
|
if (!started) return;
|
|
const config = await buildSshEnvLabFixtureConfig(started);
|
|
const spec = {
|
|
...config,
|
|
remoteCwd: started.workspaceDir,
|
|
} as const;
|
|
|
|
const preparedA = await prepareRemoteManagedRuntime({
|
|
spec,
|
|
runId: "run-commit-a",
|
|
adapterKey: "test-adapter",
|
|
workspaceLocalDir: localRepo,
|
|
});
|
|
const preparedB = await prepareRemoteManagedRuntime({
|
|
spec,
|
|
runId: "run-commit-b",
|
|
adapterKey: "test-adapter",
|
|
workspaceLocalDir: localRepo,
|
|
});
|
|
|
|
await runSshCommand(
|
|
config,
|
|
`cd ${JSON.stringify(preparedA.workspaceRemoteDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && printf "from run a\\n" > run-a.txt && git add run-a.txt && git commit -m "remote update a" >/dev/null`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
await runSshCommand(
|
|
config,
|
|
`cd ${JSON.stringify(preparedB.workspaceRemoteDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && printf "from run b\\n" > run-b.txt && git add run-b.txt && git commit -m "remote update b" >/dev/null`,
|
|
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
|
|
);
|
|
|
|
await Promise.all([
|
|
preparedA.restoreWorkspace(),
|
|
preparedB.restoreWorkspace(),
|
|
]);
|
|
|
|
await expect(readFile(path.join(localRepo, "run-a.txt"), "utf8")).resolves.toBe("from run a\n");
|
|
await expect(readFile(path.join(localRepo, "run-b.txt"), "utf8")).resolves.toBe("from run b\n");
|
|
expect(await git(localRepo, ["log", "-1", "--pretty=%s"])).toContain("Paperclip SSH sync merge");
|
|
|
|
const recentSubjects = await git(localRepo, ["log", "--pretty=%s", "-3"]);
|
|
expect(recentSubjects).toContain("remote update a");
|
|
expect(recentSubjects).toContain("remote update b");
|
|
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
|
|
});
|