Files
Devin Foley 07e98d2b2c feat(adapter-utils): add observable sandbox sync progress (#8395)
## Thinking Path

> - Paperclip is the control plane for running AI-agent companies, so
long-running remote work needs to stay observable to human operators.
> - Cloud / sandbox agents are an active roadmap area, and their
workspace sync path is part of the runtime substrate every remote coding
run depends on.
> - In the sandbox and SSH execution-target flows, Paperclip logged that
sync had started, then often went silent for the full transfer window.
> - That made large remote syncs feel stalled and also hid a real
performance problem in the command-managed sandbox upload path.
> - The first part of this pull request threads a throttled
progress-reporting surface through the adapter execution-target stack so
sync and restore work can emit meaningful updates.
> - The second part fixes the command-managed sandbox transport itself:
it removes the old serial 32KB append bottleneck, but also falls back
away from the single-stream path when a provider-backed sandbox runner
cannot surface mid-flight stdin progress.
> - The result is that sandbox and SSH transfers are both faster and
more observable, including the live Daytona-style sandbox case that
previously only emitted `0%` and `100%`.

## Linked Issues or Issue Description

No public GitHub issue exists for this bug, so it is described inline
below following the bug report template.

### What happened

- Remote sandbox and SSH workspace syncs could spend a long time
transferring data while only logging a start line (`Syncing workspace
and runtime assets to sandbox environment`) and, at best, a terminal
line.
- In the command-managed sandbox path, the original upload
implementation also paid a large performance cost by appending base64
data in many small sequential remote writes (thousands of serial 32KB
round-trips on a large workspace).
- After the initial transport rewrite, live provider-backed sandbox runs
still only emitted `0%` and `100%` because the single-stream stdin RPC
buffered progress until completion.

### Expected behavior

- Long-running sandbox and SSH syncs should periodically report how much
of the transfer is complete (a percentage and/or MB transferred) so an
operator can tell the run is healthy and making progress rather than
stuck.
- The main sandbox upload path should not be artificially slow.
- A transfer that fails partway should leave an explicit failure marker
in the log rather than a dangling intermediate percentage.

### Steps to reproduce

1. Run an agent against a sandbox (command-managed) or SSH
(remote-managed) execution target with a non-trivial workspace.
2. Watch the run log during the workspace/runtime asset sync phase.
3. Observe that the log shows the sync start line and then stays silent
for the full transfer (live provider-backed sandbox runs only show `0%`
then `100%`).

### Paperclip version or commit

- Branch `PAPA-825-provide-status-updates-when-syncing-sandboxes` off
`master`.

### Deployment mode

- Self-hosted / local instance using sandbox (command-managed) and SSH
(remote-managed) execution targets, including provider-backed sandbox
runners.

## What Changed

- Added shared throttled runtime progress reporting and threaded
`onProgress` through the adapter execution-target surface and adapter
`execute.ts` entrypoints.
- Added sync and restore progress reporting for the command-managed
sandbox path and the SSH/remote-managed path, including git
import/export progress where totals are known.
- Reworked command-managed sandbox transfer behavior so uploads use the
faster single-stream path when appropriate, but fall back to chunked
progress-emitting writes when the runner cannot expose mid-stream stdin
progress.
- Marked provider-backed environment sandbox runners as not supporting
single-stream stdin progress so live sandbox runs emit meaningful
intermediate updates instead of only `0%` and `100%`.
- Emit an explicit terminal failure marker (`failed at NN% (x/y MB)`)
when an SSH/tar transfer rejects, so a failed sync no longer leaves a
dangling intermediate percentage in the log.
- Run the SSH sync/restore size estimate (local directory walk / remote
`du` probe) concurrently with the transfer instead of awaiting it before
opening the pipe, so progress instrumentation no longer adds startup
latency proportional to workspace file count.
- Added and extended focused regression coverage for runtime progress
throttling and the new failure marker, command-managed sandbox
transfers, sandbox orchestration, SSH transfer progress, and environment
execution-target wiring.

## Verification

- `pnpm exec vitest run
packages/adapter-utils/src/runtime-progress.test.ts
packages/adapter-utils/src/ssh-fixture.test.ts
packages/adapter-utils/src/command-managed-runtime.test.ts
packages/adapter-utils/src/sandbox-managed-runtime.test.ts`
- `pnpm exec vitest run
packages/adapter-utils/src/command-managed-runtime.test.ts
server/src/__tests__/environment-execution-target.test.ts`
- `npx tsc --noEmit` for `packages/adapter-utils`

## Risks

- The provider-backed sandbox fallback now prefers chunked
command-managed writes when progress hooks are active, so
small-to-medium uploads may trade some raw throughput for observable
intermediate progress on runtimes that cannot surface true mid-stream
stdin progress.
- Progress percentages on tar-based transfers still depend on estimates
in some cases, so operators may briefly see MB-only lines before the
estimate resolves, then near-final clamping before the terminal `100%`
line.
- This PR changes shared execution-target behavior used by multiple
adapters, so regressions would most likely appear in remote runtime
setup/teardown flows rather than in a single adapter.

## Model Used

- Initial implementation: OpenAI GPT-5.4 via Codex local agent
(`codex_local`), high reasoning mode.
- Observability follow-ups (failure marker, concurrent size estimate,
added tests): Claude Opus 4.8 via Claude Code (`claude_local`).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-06-20 13:03:42 -07:00

739 lines
30 KiB
TypeScript

import { execFile } from "node:child_process";
import { mkdir, mkdtemp, readFile, rm, symlink, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it } from "vitest";
import {
buildSshSpawnTarget,
buildSshEnvLabFixtureConfig,
getSshEnvLabSupport,
prepareWorkspaceForSshExecution,
readSshEnvLabFixtureStatus,
restoreWorkspaceFromSshExecution,
runSshCommand,
syncDirectoryFromSsh,
syncDirectoryToSsh,
startSshEnvLabFixture,
stopSshEnvLabFixture,
} from "./ssh.js";
import { prepareRemoteManagedRuntime } from "./remote-managed-runtime.js";
const SSH_FIXTURE_TEST_TIMEOUT_MS = 30_000;
let sshEnvLabUnsupportedReason: string | null = null;
async function git(cwd: string, args: string[]): Promise<string> {
return await new Promise((resolve, reject) => {
execFile("git", ["-C", cwd, ...args], (error, stdout, stderr) => {
if (error) {
reject(new Error((stderr || stdout || error.message).trim()));
return;
}
resolve(stdout.trim());
});
});
}
async function startSshEnvLabFixtureOrSkip(statePath: string, label: string) {
if (sshEnvLabUnsupportedReason) {
console.warn(`Skipping ${label}: ${sshEnvLabUnsupportedReason}`);
return null;
}
const support = await getSshEnvLabSupport();
if (!support.supported) {
sshEnvLabUnsupportedReason = support.reason ?? "unsupported environment";
console.warn(`Skipping ${label}: ${sshEnvLabUnsupportedReason}`);
return null;
}
try {
return await startSshEnvLabFixture({ statePath });
} catch (error) {
sshEnvLabUnsupportedReason = error instanceof Error ? error.message : String(error);
console.warn(`Skipping ${label}: ${sshEnvLabUnsupportedReason}`);
return null;
}
}
interface ParsedProgressLine {
raw: string;
percent: number | null;
doneMb: number | null;
totalMb: number | null;
}
function parseProgressLine(line: string): ParsedProgressLine {
const trimmed = line.trimEnd();
const percentMatch = trimmed.match(/:\s*(\d+)%\s*\(([\d.]+)\/([\d.]+) MB\)$/);
if (percentMatch) {
return {
raw: trimmed,
percent: Number.parseInt(percentMatch[1]!, 10),
doneMb: Number.parseFloat(percentMatch[2]!),
totalMb: Number.parseFloat(percentMatch[3]!),
};
}
const mbMatch = trimmed.match(/:\s*([\d.]+) MB$/);
if (mbMatch) {
return { raw: trimmed, percent: null, doneMb: Number.parseFloat(mbMatch[1]!), totalMb: null };
}
return { raw: trimmed, percent: null, doneMb: null, totalMb: null };
}
describe("ssh env-lab fixture", () => {
const cleanupDirs: string[] = [];
afterEach(async () => {
while (cleanupDirs.length > 0) {
const dir = cleanupDirs.pop();
if (!dir) continue;
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
}
});
it("starts an isolated sshd fixture and executes commands through it", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH env-lab fixture test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const quotedWorkspace = JSON.stringify(started.workspaceDir);
const result = await runSshCommand(
config,
`cd ${quotedWorkspace} && pwd`,
);
expect(result.stdout.trim()).toBe(started.workspaceDir);
const status = await readSshEnvLabFixtureStatus(statePath);
expect(status.running).toBe(true);
await stopSshEnvLabFixture(statePath);
const stopped = await readSshEnvLabFixtureStatus(statePath);
expect(stopped.running).toBe(false);
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("forwards stdin to remote SSH commands", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH stdin forwarding test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const remotePath = path.posix.join(started.workspaceDir, "stdin-forwarded.txt");
await runSshCommand(
config,
`cat > ${JSON.stringify(remotePath)}`,
{
stdin: "hello over ssh stdin\n",
timeoutMs: 30_000,
maxBuffer: 256 * 1024,
},
);
const result = await runSshCommand(
config,
`cat ${JSON.stringify(remotePath)}`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
expect(result.stdout).toBe("hello over ssh stdin\n");
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("does not treat an unrelated reused pid as the running fixture", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH env-lab fixture test");
if (!started) return;
await stopSshEnvLabFixture(statePath);
await mkdir(path.dirname(statePath), { recursive: true });
await writeFile(
statePath,
JSON.stringify({ ...started, pid: process.pid }, null, 2),
{ mode: 0o600 },
);
const staleStatus = await readSshEnvLabFixtureStatus(statePath);
expect(staleStatus.running).toBe(false);
const restarted = await startSshEnvLabFixtureOrSkip(statePath, "SSH env-lab fixture restart test");
if (!restarted) return;
expect(restarted.pid).not.toBe(process.pid);
await stopSshEnvLabFixture(statePath);
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("rejects invalid environment variable keys when constructing SSH spawn targets", async () => {
await expect(
buildSshSpawnTarget({
spec: {
host: "ssh.example.test",
port: 22,
username: "ssh-user",
remoteCwd: "/srv/paperclip/workspace",
remoteWorkspacePath: "/srv/paperclip/workspace",
privateKey: null,
knownHosts: null,
strictHostKeyChecking: true,
},
command: "env",
args: [],
env: {
"BAD KEY": "value",
},
}),
).rejects.toThrow("Invalid SSH environment variable key: BAD KEY");
});
it("syncs a local directory into the remote fixture workspace", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localDir = path.join(rootDir, "local-overlay");
await mkdir(localDir, { recursive: true });
await writeFile(path.join(localDir, "message.txt"), "hello from paperclip\n", "utf8");
await writeFile(path.join(localDir, "._message.txt"), "should never sync\n", "utf8");
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH env-lab fixture test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const remoteDir = path.posix.join(started.workspaceDir, "overlay");
await syncDirectoryToSsh({
spec: {
...config,
remoteCwd: started.workspaceDir,
},
localDir,
remoteDir,
});
const result = await runSshCommand(
config,
`cat ${JSON.stringify(path.posix.join(remoteDir, "message.txt"))} && if [ -e ${JSON.stringify(path.posix.join(remoteDir, "._message.txt"))} ]; then echo appledouble-present; fi`,
);
expect(result.stdout).toContain("hello from paperclip");
expect(result.stdout).not.toContain("appledouble-present");
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("reports throttled upload progress with a clamped percent and terminal 100% line", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localDir = path.join(rootDir, "local-overlay");
await mkdir(localDir, { recursive: true });
// Multiple files large enough that tar emits several pipe chunks, so the
// byte counter crosses several step boundaries before the stream closes.
for (let index = 0; index < 4; index += 1) {
await writeFile(path.join(localDir, `blob-${index}.bin`), Buffer.alloc(256 * 1024, index + 1));
}
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH upload progress test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const remoteDir = path.posix.join(started.workspaceDir, "overlay-progress");
const lines: ParsedProgressLine[] = [];
await syncDirectoryToSsh({
spec: { ...config, remoteCwd: started.workspaceDir },
localDir,
remoteDir,
onProgress: (line) => {
lines.push(parseProgressLine(line));
},
progressLabel: "workspace",
});
expect(lines.length).toBeGreaterThan(0);
for (const line of lines) {
expect(line.raw).toContain("Syncing workspace to ssh");
}
// Monotonically increasing byte counts.
const doneSeries = lines.map((line) => line.doneMb ?? 0);
for (let index = 1; index < doneSeries.length; index += 1) {
expect(doneSeries[index]!).toBeGreaterThanOrEqual(doneSeries[index - 1]!);
}
// Percent clamped to <= 99% on every line emitted before the stream closed.
for (const line of lines.slice(0, -1)) {
if (line.percent != null) expect(line.percent).toBeLessThanOrEqual(99);
}
// Terminal completion line is 100% with matching done/total.
const last = lines.at(-1)!;
expect(last.percent).toBe(100);
expect(last.doneMb).toBe(last.totalMb);
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("reports restore progress with a terminal completion line", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localDir = path.join(rootDir, "local-overlay");
const restoreDir = path.join(rootDir, "restore-target");
await mkdir(localDir, { recursive: true });
await mkdir(restoreDir, { recursive: true });
for (let index = 0; index < 4; index += 1) {
await writeFile(path.join(localDir, `blob-${index}.bin`), Buffer.alloc(256 * 1024, index + 1));
}
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH restore progress test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const spec = { ...config, remoteCwd: started.workspaceDir } as const;
const remoteDir = path.posix.join(started.workspaceDir, "restore-source");
await syncDirectoryToSsh({ spec, localDir, remoteDir });
const lines: ParsedProgressLine[] = [];
await syncDirectoryFromSsh({
spec,
remoteDir,
localDir: restoreDir,
onProgress: (line) => {
lines.push(parseProgressLine(line));
},
progressLabel: "workspace",
});
expect(lines.length).toBeGreaterThan(0);
for (const line of lines) {
expect(line.raw).toContain("Restoring workspace from ssh");
}
// Terminal completion line: either an exact 100% (probe succeeded) or a
// final MB-received line (probe unavailable). Either is a valid terminal.
const last = lines.at(-1)!;
expect(last.percent === 100 || (last.percent === null && last.doneMb !== null)).toBe(true);
// The restored files round-tripped through the byte-counting transport.
await expect(readFile(path.join(restoreDir, "blob-0.bin"))).resolves.toEqual(
Buffer.alloc(256 * 1024, 1),
);
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("reports exact git-history import percentage from the known bundle size", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localRepo = path.join(rootDir, "local-workspace");
await mkdir(localRepo, { recursive: true });
await git(localRepo, ["init"]);
await git(localRepo, ["checkout", "-b", "main"]);
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(localRepo, "tracked.bin"), Buffer.alloc(256 * 1024, 7));
await git(localRepo, ["add", "tracked.bin"]);
await git(localRepo, ["commit", "-m", "initial"]);
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH git import progress test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const spec = { ...config, remoteCwd: started.workspaceDir } as const;
const lines: ParsedProgressLine[] = [];
await prepareWorkspaceForSshExecution({
spec,
localDir: localRepo,
remoteDir: started.workspaceDir,
onProgress: (line) => {
lines.push(parseProgressLine(line));
},
});
const importLines = lines.filter((line) => line.raw.includes("Importing git history to ssh"));
expect(importLines.length).toBeGreaterThan(0);
// Known bundle size -> exact percentage with no "workspace" label.
for (const line of importLines) {
expect(line.raw).not.toContain("workspace");
expect(line.percent).not.toBeNull();
}
const lastImport = importLines.at(-1)!;
expect(lastImport.percent).toBe(100);
expect(lastImport.doneMb).toBe(lastImport.totalMb);
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("can dereference local symlinks while syncing to the remote fixture", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const sourceDir = path.join(rootDir, "source");
const localDir = path.join(rootDir, "local-overlay");
await mkdir(sourceDir, { recursive: true });
await mkdir(localDir, { recursive: true });
await writeFile(path.join(sourceDir, "auth.json"), "{\"token\":\"secret\"}\n", "utf8");
await symlink(path.join(sourceDir, "auth.json"), path.join(localDir, "auth.json"));
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH symlink sync test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const remoteDir = path.posix.join(started.workspaceDir, "overlay-follow-links");
await syncDirectoryToSsh({
spec: {
...config,
remoteCwd: started.workspaceDir,
},
localDir,
remoteDir,
followSymlinks: true,
});
const result = await runSshCommand(
config,
`if [ -L ${JSON.stringify(path.posix.join(remoteDir, "auth.json"))} ]; then echo symlink; else echo regular; fi && cat ${JSON.stringify(path.posix.join(remoteDir, "auth.json"))}`,
);
expect(result.stdout).toContain("regular");
expect(result.stdout).toContain("{\"token\":\"secret\"}");
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("round-trips a git workspace through the SSH fixture", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localRepo = path.join(rootDir, "local-workspace");
await mkdir(localRepo, { recursive: true });
await git(localRepo, ["init"]);
await git(localRepo, ["checkout", "-b", "main"]);
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
await writeFile(path.join(localRepo, "._tracked.txt"), "should stay local only\n", "utf8");
await git(localRepo, ["add", "tracked.txt"]);
await git(localRepo, ["commit", "-m", "initial"]);
const originalHead = await git(localRepo, ["rev-parse", "HEAD"]);
await writeFile(path.join(localRepo, "tracked.txt"), "dirty local\n", "utf8");
await writeFile(path.join(localRepo, "untracked.txt"), "from local\n", "utf8");
const started = await startSshEnvLabFixtureOrSkip(statePath, "SSH workspace round-trip test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const spec = {
...config,
remoteCwd: started.workspaceDir,
} as const;
await prepareWorkspaceForSshExecution({
spec,
localDir: localRepo,
remoteDir: started.workspaceDir,
});
const remoteStatus = await runSshCommand(
config,
`cd ${JSON.stringify(started.workspaceDir)} && git status --short`,
);
expect(remoteStatus.stdout).toContain("M tracked.txt");
expect(remoteStatus.stdout).toContain("?? untracked.txt");
expect(remoteStatus.stdout).not.toContain("._tracked.txt");
await runSshCommand(
config,
`cd ${JSON.stringify(started.workspaceDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && git add tracked.txt untracked.txt && git commit -m "remote update" >/dev/null && printf "remote dirty\\n" > tracked.txt && printf "remote extra\\n" > remote-only.txt`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
await restoreWorkspaceFromSshExecution({
spec,
localDir: localRepo,
remoteDir: started.workspaceDir,
});
const restoredHead = await git(localRepo, ["rev-parse", "HEAD"]);
expect(restoredHead).not.toBe(originalHead);
expect(await git(localRepo, ["log", "-1", "--pretty=%s"])).toBe("remote update");
expect(await git(localRepo, ["status", "--short"])).toContain("M tracked.txt");
expect(await git(localRepo, ["status", "--short"])).not.toContain("._tracked.txt");
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("preserves both concurrent SSH restores in a shared git workspace", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localRepo = path.join(rootDir, "local-workspace");
await mkdir(localRepo, { recursive: true });
await git(localRepo, ["init"]);
await git(localRepo, ["checkout", "-b", "main"]);
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
await git(localRepo, ["add", "tracked.txt"]);
await git(localRepo, ["commit", "-m", "initial"]);
const started = await startSshEnvLabFixtureOrSkip(statePath, "concurrent SSH restore test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const spec = {
...config,
remoteCwd: started.workspaceDir,
} as const;
const preparedA = await prepareRemoteManagedRuntime({
spec,
runId: "run-a",
adapterKey: "test-adapter",
workspaceLocalDir: localRepo,
});
const preparedB = await prepareRemoteManagedRuntime({
spec,
runId: "run-b",
adapterKey: "test-adapter",
workspaceLocalDir: localRepo,
});
expect(preparedA.workspaceRemoteDir).not.toBe(preparedB.workspaceRemoteDir);
await runSshCommand(
config,
`printf "from run a\\n" > ${JSON.stringify(path.posix.join(preparedA.workspaceRemoteDir, "run-a.txt"))}`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
await runSshCommand(
config,
`printf "from run b\\n" > ${JSON.stringify(path.posix.join(preparedB.workspaceRemoteDir, "run-b.txt"))}`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
await Promise.all([
preparedA.restoreWorkspace(),
preparedB.restoreWorkspace(),
]);
await expect(readFile(path.join(localRepo, "run-a.txt"), "utf8")).resolves.toBe("from run a\n");
await expect(readFile(path.join(localRepo, "run-b.txt"), "utf8")).resolves.toBe("from run b\n");
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("preserves nested per-run files across sequential SSH restores with stale baselines", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localRepo = path.join(rootDir, "local-workspace");
await mkdir(localRepo, { recursive: true });
await git(localRepo, ["init"]);
await git(localRepo, ["checkout", "-b", "main"]);
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
await git(localRepo, ["add", "tracked.txt"]);
await git(localRepo, ["commit", "-m", "initial"]);
const started = await startSshEnvLabFixtureOrSkip(statePath, "sequential nested SSH restore test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const spec = {
...config,
remoteCwd: started.workspaceDir,
} as const;
const preparedA = await prepareRemoteManagedRuntime({
spec,
runId: "run-a",
adapterKey: "test-adapter",
workspaceLocalDir: localRepo,
});
const preparedB = await prepareRemoteManagedRuntime({
spec,
runId: "run-b",
adapterKey: "test-adapter",
workspaceLocalDir: localRepo,
});
await runSshCommand(
config,
`mkdir -p ${JSON.stringify(path.posix.join(preparedA.workspaceRemoteDir, "manual-qa/environment-matrix/ssh"))} && printf "from run a\\n" > ${JSON.stringify(path.posix.join(preparedA.workspaceRemoteDir, "manual-qa/environment-matrix/ssh/claude_local.md"))}`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
await runSshCommand(
config,
`mkdir -p ${JSON.stringify(path.posix.join(preparedB.workspaceRemoteDir, "manual-qa/environment-matrix/ssh"))} && printf "from run b\\n" > ${JSON.stringify(path.posix.join(preparedB.workspaceRemoteDir, "manual-qa/environment-matrix/ssh/codex_local.md"))}`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
await preparedA.restoreWorkspace();
await preparedB.restoreWorkspace();
await expect(readFile(path.join(localRepo, "manual-qa/environment-matrix/ssh/claude_local.md"), "utf8")).resolves
.toBe("from run a\n");
await expect(readFile(path.join(localRepo, "manual-qa/environment-matrix/ssh/codex_local.md"), "utf8")).resolves
.toBe("from run b\n");
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("round-trips remote git commits through the managed runtime restore path", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localRepo = path.join(rootDir, "local-workspace");
await mkdir(localRepo, { recursive: true });
await git(localRepo, ["init"]);
await git(localRepo, ["checkout", "-b", "main"]);
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
await git(localRepo, ["add", "tracked.txt"]);
await git(localRepo, ["commit", "-m", "initial"]);
const started = await startSshEnvLabFixtureOrSkip(statePath, "managed-runtime SSH git round-trip test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const spec = {
...config,
remoteCwd: started.workspaceDir,
} as const;
const prepared = await prepareRemoteManagedRuntime({
spec,
runId: "run-commit",
adapterKey: "test-adapter",
workspaceLocalDir: localRepo,
});
await runSshCommand(
config,
`cd ${JSON.stringify(prepared.workspaceRemoteDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && printf "committed\\n" > tracked.txt && git add tracked.txt && git commit -m "remote update" >/dev/null && printf "dirty remote\\n" > tracked.txt`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
await prepared.restoreWorkspace();
expect(await git(localRepo, ["log", "-1", "--pretty=%s"])).toBe("remote update");
await expect(readFile(path.join(localRepo, "tracked.txt"), "utf8")).resolves.toBe("dirty remote\n");
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("propagates remote commits to the local worktree with no git remote configured (no-remote-git contract)", async () => {
// Locks in the architectural contract documented in
// packages/adapter-utils/README.md and packages/adapters/AUTHORING.md:
// the local execution-workspace cwd is the only persistence boundary
// across runs. No adapter may depend on a git remote for cross-run state.
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localRepo = path.join(rootDir, "local-workspace");
await mkdir(localRepo, { recursive: true });
await git(localRepo, ["init"]);
await git(localRepo, ["checkout", "-b", "main"]);
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
await git(localRepo, ["add", "tracked.txt"]);
await git(localRepo, ["commit", "-m", "initial"]);
// Assert there is no git remote configured before we begin, and verify
// that no point in the round-trip introduces one. `git remote` returns an
// empty string when no remotes exist (and exit code 0).
expect(await git(localRepo, ["remote"])).toBe("");
const started = await startSshEnvLabFixtureOrSkip(
statePath,
"no-remote-git contract test",
);
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const spec = {
...config,
remoteCwd: started.workspaceDir,
} as const;
const prepared = await prepareRemoteManagedRuntime({
spec,
runId: "run-no-remote",
adapterKey: "test-adapter",
workspaceLocalDir: localRepo,
});
// Remote commit lands a deliverable that must show up locally via
// sync-back alone — no `git push`, no fetch from any origin.
await runSshCommand(
config,
`cd ${JSON.stringify(prepared.workspaceRemoteDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && printf "deliverable\\n" > tracked.txt && git add tracked.txt && git commit -m "remote-only commit" >/dev/null`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
await prepared.restoreWorkspace();
expect(await git(localRepo, ["log", "-1", "--pretty=%s"])).toBe(
"remote-only commit",
);
expect(await readFile(path.join(localRepo, "tracked.txt"), "utf8")).toBe(
"deliverable\n",
);
// Final assertion: still no git remote — restore did not silently add one.
expect(await git(localRepo, ["remote"])).toBe("");
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
it("merges concurrent remote commits through the managed runtime restore path", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-ssh-fixture-"));
cleanupDirs.push(rootDir);
const statePath = path.join(rootDir, "state.json");
const localRepo = path.join(rootDir, "local-workspace");
await mkdir(localRepo, { recursive: true });
await git(localRepo, ["init"]);
await git(localRepo, ["checkout", "-b", "main"]);
await git(localRepo, ["config", "user.name", "Paperclip Test"]);
await git(localRepo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(localRepo, "tracked.txt"), "base\n", "utf8");
await git(localRepo, ["add", "tracked.txt"]);
await git(localRepo, ["commit", "-m", "initial"]);
const started = await startSshEnvLabFixtureOrSkip(statePath, "concurrent managed-runtime SSH git merge test");
if (!started) return;
const config = await buildSshEnvLabFixtureConfig(started);
const spec = {
...config,
remoteCwd: started.workspaceDir,
} as const;
const preparedA = await prepareRemoteManagedRuntime({
spec,
runId: "run-commit-a",
adapterKey: "test-adapter",
workspaceLocalDir: localRepo,
});
const preparedB = await prepareRemoteManagedRuntime({
spec,
runId: "run-commit-b",
adapterKey: "test-adapter",
workspaceLocalDir: localRepo,
});
await runSshCommand(
config,
`cd ${JSON.stringify(preparedA.workspaceRemoteDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && printf "from run a\\n" > run-a.txt && git add run-a.txt && git commit -m "remote update a" >/dev/null`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
await runSshCommand(
config,
`cd ${JSON.stringify(preparedB.workspaceRemoteDir)} && git config user.name "Paperclip SSH" && git config user.email "ssh@paperclip.dev" && printf "from run b\\n" > run-b.txt && git add run-b.txt && git commit -m "remote update b" >/dev/null`,
{ timeoutMs: 30_000, maxBuffer: 256 * 1024 },
);
await Promise.all([
preparedA.restoreWorkspace(),
preparedB.restoreWorkspace(),
]);
await expect(readFile(path.join(localRepo, "run-a.txt"), "utf8")).resolves.toBe("from run a\n");
await expect(readFile(path.join(localRepo, "run-b.txt"), "utf8")).resolves.toBe("from run b\n");
expect(await git(localRepo, ["log", "-1", "--pretty=%s"])).toContain("Paperclip SSH sync merge");
const recentSubjects = await git(localRepo, ["log", "--pretty=%s", "-3"]);
expect(recentSubjects).toContain("remote update a");
expect(recentSubjects).toContain("remote update b");
}, SSH_FIXTURE_TEST_TIMEOUT_MS);
});