Fix sandbox git publishing and large workspace uploads (#8422)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Sandbox and SSH runtimes need to preserve agent work across isolated
execution environments
> - Git-backed workspaces were being copied mostly as filesystem
archives, which breaks when `.git` points outside the mounted workspace
and makes sandbox agents unable to publish their own branches
> - Large ignored dependency trees could also be swept into the sandbox
overlay, causing multi-GB transfers and max-string failures in some
sandbox clients
> - This pull request makes sandbox runtime setup use a git-backed HEAD
sync plus a small dirty/untracked overlay, and bounds sandbox file
transfers so large archives do not need one huge string
> - The benefit is that sandbox agents can commit and push from a usable
git checkout without uploading dependency trees such as `node_modules`

## Linked Issues or Issue Description

Refs #8395

No public duplicate issue or PR was found after searches for `sandbox
git workspace`, `git push sandbox`, and `node_modules sandbox upload`.

Bug report:

- What happened: sandbox-backed agent workspaces could receive a `.git`
file that pointed at host-only git state, leaving the sandbox unable to
run normal git workflows. The sandbox overlay upload could also include
ignored dependency directories, creating very large transfers.
- Expected behavior: sandbox and remote runtimes should prepare a usable
git-backed workspace, copy only the necessary workspace overlay, and
restore git history plus file changes without depending on a host-only
`.git` path.
- Steps to reproduce:
1. Run an agent in a sandbox-backed workspace whose local git checkout
is a worktree.
2. Ask the agent to complete a GitHub workflow that requires commit/push
access.
3. Observe that git operations can fail inside the sandbox, and ignored
dependency trees can be uploaded as part of the workspace overlay.
- Paperclip version or commit: reproduced against `master` before this
PR, base `7aa212296eb1`.
- Deployment mode: local dev / sandbox-backed runtime.
- Installation method: built from source.
- Agent adapters involved: local adapters using shared adapter-utils
runtime preparation.
- Database mode: not database-related.
- Access context: agent runtime.
- Local verification environment: Node.js v25.6.1, pnpm 9.15.4, macOS
arm64.
- Privacy checklist: all pasted output was reviewed for secrets, private
hostnames, local usernames, and internal instance links.

## What Changed

- Added a GitHub workflow push preflight so agent runs can detect
missing push credentials when a workflow explicitly needs GitHub
publishing.
- Added shared git workspace sync helpers for shallow HEAD import/export
and dirty/untracked overlay tracking.
- Updated sandbox managed runtime setup to use git history plus a
selected overlay instead of uploading the full local workspace for
git-backed workspaces.
- Bounded sandbox archive upload/download paths so large payloads stream
or chunk instead of materializing one oversized string.
- Excluded `.git` and ignored dependency trees from sandbox upload,
download, and restore baselines while preserving local ignored
directories during sync-back.
- Added focused tests for git workspace sync, sandbox overlay selection,
transfer chunking, and heartbeat push-preflight behavior.

## Verification

- `pnpm exec vitest run
packages/adapter-utils/src/git-workspace-sync.test.ts
packages/adapter-utils/src/sandbox-managed-runtime.test.ts
packages/adapter-utils/src/command-managed-runtime.test.ts
server/src/__tests__/heartbeat-project-env.test.ts
server/src/__tests__/heartbeat-workspace-session.test.ts`
- `pnpm --filter @paperclipai/adapter-utils typecheck`
- `pnpm --filter @paperclipai/server typecheck`
- `pnpm build`
- Public-hygiene scan of the PR diff and commit messages for internal
issue ids, local paths, private hostnames, and obvious token patterns.

## Risks

- Medium risk: this changes sandbox runtime synchronization semantics
for git-backed workspaces, especially around dirty tracked files,
untracked files, deleted paths, and ignored files.
- The main mitigation is focused test coverage for upload contents,
restore exclusions, and git round-trip behavior.
- The SSH runtime keeps the current bundle-based implementation from
`master`; this PR only aligns shared excludes and sandbox behavior with
that model.

## Model Used

OpenAI Codex, GPT-5-based coding agent, tool-enabled shell/git/GitHub
workflow, with code execution and repository inspection.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
2026-06-20 22:03:55 -07:00
committed by GitHub
parent 33353ce62b
commit 2c98c8e1e5
13 changed files with 1416 additions and 143 deletions
@@ -22,7 +22,10 @@ interface SpawnRunnerHandle {
// A runner that actually executes the shell scripts (piping stdin through a real
// pipe so multi-MB payloads work) and replays stdout through onLog in several
// chunks so the streaming readFile byte-counter is exercised.
function makeSpawnRunner(options: { supportsSingleStreamStdinProgress?: boolean } = {}): SpawnRunnerHandle {
function makeSpawnRunner(options: {
supportsSingleStreamStdinProgress?: boolean;
maxStdoutBytes?: number;
} = {}): SpawnRunnerHandle {
const calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }> = [];
const runner: CommandManagedRuntimeRunner = {
supportsSingleStreamStdinProgress: options.supportsSingleStreamStdinProgress,
@@ -48,6 +51,21 @@ function makeSpawnRunner(options: { supportsSingleStreamStdinProgress?: boolean
resolve({ exitCode: 127, signal: null, timedOut: false, stdout, stderr, pid: null, startedAt });
});
child.on("close", async (code) => {
if (
options.maxStdoutBytes != null &&
Buffer.byteLength(stdout, "utf8") > options.maxStdoutBytes
) {
resolve({
exitCode: 1,
signal: null,
timedOut: false,
stdout,
stderr: `stdout exceeded ${options.maxStdoutBytes} bytes`,
pid: child.pid ?? null,
startedAt,
});
return;
}
if (input.onLog && stdout.length > 0) {
const chunkSize = Math.max(1, Math.ceil(stdout.length / 4));
for (let offset = 0; offset < stdout.length; offset += chunkSize) {
@@ -75,6 +93,26 @@ function toArrayBuffer(buffer: Buffer): ArrayBuffer {
return Uint8Array.from(buffer).buffer;
}
async function withBase64StringByteLimit<T>(limitBytes: number, fn: () => Promise<T>): Promise<T> {
const originalToString = Buffer.prototype.toString;
Buffer.prototype.toString = function patchedToString(
this: Buffer,
encoding?: BufferEncoding,
start?: number,
end?: number,
) {
if (encoding === "base64" && this.byteLength > limitBytes) {
throw new Error(`test guard: attempted to base64-encode ${this.byteLength} bytes at once`);
}
return originalToString.call(this, encoding, start, end);
} as typeof Buffer.prototype.toString;
try {
return await fn();
} finally {
Buffer.prototype.toString = originalToString;
}
}
describe("command managed runtime", () => {
const cleanupDirs: string[] = [];
@@ -243,10 +281,12 @@ describe("command managed runtime", () => {
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
const progress: Array<{ done: number; total: number | null }> = [];
await client.writeFile(remotePath, toArrayBuffer(payload), {
onProgress: (done, total) => {
progress.push({ done, total });
},
await withBase64StringByteLimit(4 * 1024 * 1024, async () => {
await client.writeFile(remotePath, toArrayBuffer(payload), {
onProgress: (done, total) => {
progress.push({ done, total });
},
});
});
// Exactly one upload process: O(1) round-trips regardless of payload size.
@@ -288,6 +328,9 @@ describe("command managed runtime", () => {
// Provider-backed sandbox runners cannot surface mid-flight progress for a
// single stdin RPC, so we intentionally use several large append commands.
expect(calls.length).toBeGreaterThan(2);
const stdinCalls = calls.filter((call) => call.stdin != null);
expect(stdinCalls.length).toBeGreaterThan(2);
expect(stdinCalls.every((call) => Buffer.byteLength(call.stdin ?? "", "utf8") <= 4.1 * 1024 * 1024)).toBe(true);
expect(progress.length).toBeGreaterThan(2);
for (let i = 1; i < progress.length; i++) {
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
@@ -295,16 +338,41 @@ describe("command managed runtime", () => {
expect(progress.at(-1)).toEqual({ done: payload.length, total: payload.length });
});
it("streams a download through onLog and reports monotonic byte progress to the total", async () => {
it("falls back to bounded chunks when the runner does not explicitly opt in", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-write-fallback-no-progress-"));
cleanupDirs.push(rootDir);
const remotePath = path.join(rootDir, "nested", "payload.bin");
const payload = Buffer.alloc(12 * 1024 * 1024);
for (let i = 0; i < payload.length; i++) payload[i] = i % 256;
const { runner, calls } = makeSpawnRunner();
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
await withBase64StringByteLimit(4 * 1024 * 1024, async () => {
await client.writeFile(remotePath, toArrayBuffer(payload));
});
const written = await readFile(remotePath);
expect(written.equals(payload)).toBe(true);
// A runner that doesn't mark single-stream stdin support must avoid passing
// the whole base64 archive as one string, so we expect multiple append calls.
const stdinCalls = calls.filter((call) => call.stdin != null);
expect(stdinCalls.length).toBeGreaterThan(1);
expect(stdinCalls.every((call) => Buffer.byteLength(call.stdin ?? "", "utf8") <= 4.1 * 1024 * 1024)).toBe(true);
});
it("downloads in bounded stdout chunks and reports monotonic byte progress to the total", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-read-"));
cleanupDirs.push(rootDir);
const remotePath = path.join(rootDir, "download.bin");
const payload = Buffer.alloc(512 * 1024);
const payload = Buffer.alloc(7 * 1024 * 1024);
for (let i = 0; i < payload.length; i++) payload[i] = (i * 7) % 256;
await writeFile(remotePath, payload);
const { runner } = makeSpawnRunner();
const { runner, calls } = makeSpawnRunner({ maxStdoutBytes: 5 * 1024 * 1024 });
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
const progress: Array<{ done: number; total: number | null }> = [];
@@ -316,7 +384,10 @@ describe("command managed runtime", () => {
expect(Buffer.from(bytes as ArrayBuffer).equals(payload)).toBe(true);
// The runner replays stdout in several chunks, so progress arrives in steps.
// The old single `base64 < file` path would exceed the runner's stdout cap.
// The bounded path reads with several small `dd | base64` commands instead.
expect(calls.some((call) => call.args?.join(" ").includes("base64 <"))).toBe(false);
expect(calls.filter((call) => call.args?.join(" ").includes("dd if=")).length).toBeGreaterThan(1);
expect(progress.length).toBeGreaterThan(1);
for (let i = 1; i < progress.length; i++) {
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
@@ -56,6 +56,12 @@ const REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES = 96 * 1024 * 1024;
// Fallback chunk size (base64 bytes). Kept a multiple of 4 so each chunk is a
// self-contained base64 unit that decodes cleanly on its own.
const REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE = 4 * 1024 * 1024;
const REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE = (REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE / 4) * 3;
const REMOTE_READ_CHUNK_BYTES = REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE;
function base64EncodedLength(byteLength: number): number {
return Math.ceil(byteLength / 3) * 4;
}
function toBuffer(bytes: Buffer | Uint8Array | ArrayBuffer): Buffer {
if (Buffer.isBuffer(bytes)) return bytes;
@@ -104,11 +110,10 @@ export function createCommandManagedRuntimeClient(input: {
writeFile: async (remotePath, bytes, options) => {
const buffer = toBuffer(bytes);
const total = buffer.byteLength;
const body = buffer.toString("base64");
const encodedLength = base64EncodedLength(total);
const remoteDir = path.posix.dirname(remotePath);
const remoteTempPath = `${remotePath}.paperclip-upload`;
const canUseSingleStreamProgressPath =
!options?.onProgress || input.runner.supportsSingleStreamStdinProgress === true;
const canUseSingleStreamProgressPath = input.runner.supportsSingleStreamStdinProgress === true;
// Primary path: a single round-trip. Stream the entire base64 body to one
// `base64 -d` process via stdin, decode straight into a temp file, then
@@ -116,9 +121,10 @@ export function createCommandManagedRuntimeClient(input: {
// one `printf >> tmpfile` shell round-trip per 32 KB — thousands of serial
// processes for a large workspace — with exactly one process.
if (
body.length <= REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES &&
encodedLength <= REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES &&
canUseSingleStreamProgressPath
) {
const body = buffer.toString("base64");
await options?.onProgress?.(0, total);
await runShell(
`mkdir -p ${shellQuote(remoteDir)} && ` +
@@ -139,62 +145,50 @@ export function createCommandManagedRuntimeClient(input: {
`mkdir -p ${shellQuote(remoteDir)} && ` +
`rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`,
);
for (let offset = 0; offset < body.length; offset += REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE) {
const chunk = body.slice(offset, offset + REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE);
for (let offset = 0; offset < total; offset += REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE) {
const end = Math.min(total, offset + REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE);
const chunk = buffer.subarray(offset, end).toString("base64");
await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk });
const decodedSoFar = Math.min(total, Math.floor(((offset + chunk.length) * 3) / 4));
await options?.onProgress?.(decodedSoFar, total);
await options?.onProgress?.(end, total);
}
await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`);
await options?.onProgress?.(total, total);
},
readFile: async (remotePath, options) => {
// Decoded file size up front so download progress can be a percentage.
// Only paid when a progress hook is attached.
let totalBytes: number | null = null;
if (options?.onProgress) {
const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`);
const parsed = Number.parseInt(sizeResult.stdout.trim(), 10);
totalBytes = Number.isFinite(parsed) && parsed >= 0 ? parsed : null;
// Chunked reads intentionally query the remote size first, even without
// a progress sink, so each sandbox RPC stays bounded and truncation is
// detected without materializing the whole file as one stdout string.
const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`);
const totalBytes = Number.parseInt(sizeResult.stdout.trim(), 10);
if (!Number.isFinite(totalBytes) || totalBytes < 0) {
throw new Error(`Could not determine remote file size for ${remotePath}`);
}
// Stream the remote `base64` stdout through a byte counter, decoding each
// 4-char-aligned slice incrementally rather than buffering the whole
// response as one string. Falls back to the buffered result when the
// runner does not surface incremental stdout.
// Read in bounded remote chunks so the runner never has to materialize a
// single base64 stdout string for the whole archive. The client API still
// returns the decoded file as a Buffer, but every command result stays
// small enough for provider-backed sandbox RPCs.
const decodedChunks: Buffer[] = [];
let b64Remainder = "";
let decodedSoFar = 0;
let streamed = false;
const result = await runShell(`base64 < ${shellQuote(remotePath)}`, {
onLog: async (stream, chunk) => {
if (stream !== "stdout") return;
streamed = true;
const data = b64Remainder + chunk.replace(/\s+/g, "");
const alignedLen = data.length - (data.length % 4);
if (alignedLen > 0) {
const decoded = Buffer.from(data.slice(0, alignedLen), "base64");
decodedChunks.push(decoded);
decodedSoFar += decoded.byteLength;
}
b64Remainder = data.slice(alignedLen);
if (options?.onProgress) {
const done = totalBytes != null ? Math.min(totalBytes, decodedSoFar) : decodedSoFar;
await options.onProgress(done, totalBytes);
}
},
});
let out: Buffer;
if (streamed) {
if (b64Remainder.length > 0) {
decodedChunks.push(Buffer.from(b64Remainder, "base64"));
}
out = Buffer.concat(decodedChunks);
} else {
out = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
if (totalBytes === 0) {
await options?.onProgress?.(0, 0);
return Buffer.alloc(0);
}
await options?.onProgress?.(out.byteLength, totalBytes ?? out.byteLength);
for (let chunkIndex = 0; decodedSoFar < totalBytes; chunkIndex++) {
const result = await runShell(
`dd if=${shellQuote(remotePath)} bs=${REMOTE_READ_CHUNK_BYTES} skip=${chunkIndex} count=1 2>/dev/null | base64`,
);
const chunk = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
if (chunk.byteLength === 0) break;
decodedChunks.push(chunk);
decodedSoFar += chunk.byteLength;
await options?.onProgress?.(Math.min(decodedSoFar, totalBytes), totalBytes);
}
const out = Buffer.concat(decodedChunks);
if (out.byteLength !== totalBytes) {
throw new Error(`Remote file read was truncated for ${remotePath}: ${out.byteLength}/${totalBytes} bytes`);
}
await options?.onProgress?.(out.byteLength, totalBytes);
return out;
},
listFiles: async (remotePath) => {
@@ -0,0 +1,28 @@
export function isRelativePathOrDescendant(relative: string, candidate: string): boolean {
return relative === candidate || relative.startsWith(`${candidate}/`);
}
function pathContainsSegmentOrDescendant(relative: string, segment: string): boolean {
return relative === segment ||
relative.startsWith(`${segment}/`) ||
relative.endsWith(`/${segment}`) ||
relative.includes(`/${segment}/`);
}
export function excludePatternMatches(relative: string, pattern: string): boolean {
if (pattern.startsWith("*/") && pattern.endsWith("/*")) {
return pathContainsSegmentOrDescendant(relative, pattern.slice(2, -2));
}
if (pattern.startsWith("*/")) {
return pathContainsSegmentOrDescendant(relative, pattern.slice(2));
}
if (pattern.endsWith("/*")) {
const base = pattern.slice(0, -2);
return relative.startsWith(`${base}/`);
}
return isRelativePathOrDescendant(relative, pattern);
}
export function shouldExcludePath(relative: string, exclude: readonly string[]): boolean {
return exclude.some((entry) => excludePatternMatches(relative, entry));
}
@@ -0,0 +1,128 @@
import { execFile as execFileCallback } from "node:child_process";
import { lstat, mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import { afterEach, describe, expect, it } from "vitest";
import {
buildRemoteGitDeltaBundleScript,
createImportedGitRef,
createRemoteGitExportRef,
deleteLocalGitRef,
fetchGitBundleIntoLocalRef,
readGitWorkspaceSnapshot,
runLocalGit,
withShallowGitWorkspaceClone,
} from "./git-workspace-sync.js";
const execFile = promisify(execFileCallback);
async function git(cwd: string, args: string[]): Promise<string> {
return (await runLocalGit(cwd, args)).stdout.trim();
}
describe("git workspace sync", () => {
const cleanupDirs: string[] = [];
afterEach(async () => {
while (cleanupDirs.length > 0) {
const dir = cleanupDirs.pop();
if (!dir) continue;
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
}
});
async function createRepo(rootDir: string): Promise<string> {
const repo = path.join(rootDir, "repo");
await mkdir(repo, { recursive: true });
await git(repo, ["init"]);
await git(repo, ["checkout", "-b", "main"]);
await git(repo, ["config", "user.name", "Paperclip Test"]);
await git(repo, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
await git(repo, ["add", "tracked.txt"]);
await git(repo, ["commit", "-m", "base"]);
return repo;
}
it("creates a shallow standalone clone from the local HEAD snapshot", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-sync-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
await rm(path.join(repo, "tracked.txt"));
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot).toMatchObject({
headCommit: baseHead,
branchName: "main",
deletedPaths: ["tracked.txt"],
});
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (cloneDir) => {
expect((await lstat(path.join(cloneDir, ".git"))).isDirectory()).toBe(true);
await expect(readFile(path.join(cloneDir, ".git", "shallow"), "utf8")).resolves.toContain(baseHead);
expect(await git(cloneDir, ["rev-list", "--count", "HEAD"])).toBe("1");
expect(await git(cloneDir, ["branch", "--show-current"])).toBe("main");
await expect(readFile(path.join(cloneDir, "tracked.txt"), "utf8")).resolves.toBe("base\n");
});
});
it("builds thin git delta bundles relative to the imported base", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-delta-"));
cleanupDirs.push(rootDir);
const repo = await createRepo(rootDir);
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
const snapshot = await readGitWorkspaceSnapshot(repo);
expect(snapshot).not.toBeNull();
await withShallowGitWorkspaceClone({
localDir: repo,
snapshot: snapshot!,
}, async (remoteDir) => {
const emptyBundle = path.join(rootDir, "empty.bundle");
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir,
baseSha: baseHead,
exportRef: createRemoteGitExportRef("test"),
bundlePath: emptyBundle,
})]);
expect((await stat(emptyBundle)).size).toBe(0);
await git(remoteDir, ["config", "user.name", "Paperclip Remote"]);
await git(remoteDir, ["config", "user.email", "remote@paperclip.dev"]);
await writeFile(path.join(remoteDir, "tracked.txt"), "remote\n", "utf8");
await git(remoteDir, ["commit", "-am", "remote update"]);
const remoteHead = await git(remoteDir, ["rev-parse", "HEAD"]);
const deltaBundle = path.join(rootDir, "delta.bundle");
const importedRef = createImportedGitRef("test");
const exportRef = createRemoteGitExportRef("test");
try {
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir,
baseSha: baseHead,
exportRef,
bundlePath: deltaBundle,
})]);
expect((await stat(deltaBundle)).size).toBeGreaterThan(0);
const importedHead = await fetchGitBundleIntoLocalRef({
localDir: repo,
bundlePath: deltaBundle,
exportRef,
importedRef,
baseSha: baseHead,
});
expect(importedHead).toBe(remoteHead);
expect(await git(repo, ["rev-list", "--count", importedRef, "--not", baseHead])).toBe("1");
} finally {
await deleteLocalGitRef({ localDir: repo, ref: importedRef });
}
});
});
});
@@ -0,0 +1,321 @@
import { randomUUID } from "node:crypto";
import { execFile } from "node:child_process";
import { promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
export interface GitCommandResult {
stdout: string;
stderr: string;
}
export interface GitWorkspaceSnapshot {
headCommit: string;
branchName: string | null;
overlayPaths: string[];
deletedPaths: string[];
ignoredPaths: string[];
}
export const GIT_ARCHIVE_EXCLUDES = [".git", ".git/*"] as const;
function shellQuote(value: string) {
return `'${value.replace(/'/g, `'\"'\"'`)}'`;
}
export async function runLocalGit(
localDir: string,
args: string[],
options: {
timeout?: number;
maxBuffer?: number;
} = {},
): Promise<GitCommandResult> {
return await new Promise<GitCommandResult>((resolve, reject) => {
execFile(
"git",
["-C", localDir, ...args],
{
timeout: options.timeout ?? 15_000,
maxBuffer: options.maxBuffer ?? 1024 * 128,
},
(error, stdout, stderr) => {
if (error) {
reject(Object.assign(error, { stdout: stdout ?? "", stderr: stderr ?? "" }));
return;
}
resolve({
stdout: stdout ?? "",
stderr: stderr ?? "",
});
},
);
});
}
export async function readGitWorkspaceSnapshot(localDir: string): Promise<GitWorkspaceSnapshot | null> {
try {
const insideWorkTree = await runLocalGit(localDir, ["rev-parse", "--is-inside-work-tree"], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
if (insideWorkTree.stdout.trim() !== "true") {
return null;
}
const [headCommitResult, branchResult, overlayDiffResult, untrackedResult, deletedResult, ignoredResult] = await Promise.all([
runLocalGit(localDir, ["rev-parse", "HEAD"], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}),
runLocalGit(localDir, ["rev-parse", "--abbrev-ref", "HEAD"], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}),
runLocalGit(localDir, ["diff", "--name-only", "-z", "--diff-filter=ACMRTUXB", "HEAD", "--"], {
timeout: 10_000,
maxBuffer: 1024 * 1024,
}),
runLocalGit(localDir, ["ls-files", "--others", "--exclude-standard", "-z"], {
timeout: 10_000,
maxBuffer: 1024 * 1024,
}),
runLocalGit(localDir, ["diff", "--name-only", "-z", "--diff-filter=D", "HEAD", "--"], {
timeout: 10_000,
maxBuffer: 256 * 1024,
}),
runLocalGit(localDir, ["status", "--ignored", "--porcelain=v1", "-z", "--untracked-files=normal"], {
timeout: 10_000,
maxBuffer: 1024 * 1024,
}),
]);
const branchName = branchResult.stdout.trim();
const splitNul = (value: string) => value.split("\0").map((entry) => entry.trim()).filter(Boolean);
return {
headCommit: headCommitResult.stdout.trim(),
branchName: branchName && branchName !== "HEAD" ? branchName : null,
overlayPaths: [...new Set([...splitNul(overlayDiffResult.stdout), ...splitNul(untrackedResult.stdout)])]
.sort((left, right) => left.localeCompare(right)),
deletedPaths: [...new Set(splitNul(deletedResult.stdout))]
.sort((left, right) => left.localeCompare(right)),
ignoredPaths: splitNul(ignoredResult.stdout)
.filter((entry) => entry.startsWith("!! "))
.map((entry) => entry.slice(3).replace(/\/+$/, ""))
.filter(Boolean)
.sort((left, right) => left.localeCompare(right)),
};
} catch {
return null;
}
}
export async function withShallowGitWorkspaceClone<T>(
input: {
localDir: string;
snapshot: GitWorkspaceSnapshot;
},
fn: (cloneDir: string) => Promise<T>,
): Promise<T> {
const cloneDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-workspace-"));
const tempRef = `refs/paperclip/git-sync/import/${randomUUID()}`;
try {
await runLocalGit(input.localDir, ["update-ref", tempRef, input.snapshot.headCommit], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
await runLocalGit(cloneDir, ["init"], {
timeout: 10_000,
maxBuffer: 64 * 1024,
});
await runLocalGit(cloneDir, ["fetch", "--depth=1", input.localDir, tempRef], {
timeout: 60_000,
maxBuffer: 1024 * 1024,
});
await runLocalGit(
cloneDir,
input.snapshot.branchName
? ["checkout", "--force", "-B", input.snapshot.branchName, "FETCH_HEAD"]
: ["checkout", "--force", "--detach", "FETCH_HEAD"],
{
timeout: 60_000,
maxBuffer: 1024 * 1024,
},
);
await runLocalGit(cloneDir, ["reset", "--hard", input.snapshot.headCommit], {
timeout: 60_000,
maxBuffer: 1024 * 1024,
});
return await fn(cloneDir);
} finally {
await runLocalGit(input.localDir, ["update-ref", "-d", tempRef], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}).catch(() => undefined);
await fs.rm(cloneDir, { recursive: true, force: true }).catch(() => undefined);
}
}
export function createImportedGitRef(scope = "remote"): string {
return `refs/paperclip/git-sync/imported/${scope}/${randomUUID()}`;
}
export function createRemoteGitExportRef(scope = "remote"): string {
return `refs/paperclip/git-sync/export/${scope}/${randomUUID()}`;
}
export async function deleteLocalGitRef(input: {
localDir: string;
ref: string;
}): Promise<void> {
await runLocalGit(input.localDir, ["update-ref", "-d", input.ref], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}).catch(() => undefined);
}
export async function fetchGitBundleIntoLocalRef(input: {
localDir: string;
bundlePath: string;
exportRef: string;
importedRef: string;
baseSha: string;
}): Promise<string> {
const bundleSize = (await fs.stat(input.bundlePath).catch(() => null))?.size ?? 0;
if (bundleSize === 0) {
return input.baseSha;
}
await runLocalGit(input.localDir, ["fetch", "--force", input.bundlePath, `${input.exportRef}:${input.importedRef}`], {
timeout: 60_000,
maxBuffer: 1024 * 1024,
});
const importedHead = await runLocalGit(input.localDir, ["rev-parse", input.importedRef], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
return importedHead.stdout.trim();
}
export function buildRemoteGitDeltaBundleScript(input: {
remoteDir: string;
baseSha: string;
exportRef: string;
bundlePath: string;
catBundle?: boolean;
cleanupBundle?: boolean;
}): string {
const remoteDir = shellQuote(input.remoteDir);
const bundlePath = shellQuote(input.bundlePath);
const exportRef = shellQuote(input.exportRef);
const baseSha = shellQuote(input.baseSha);
const cleanupParts = [
`rm -f ${bundlePath}`,
`git -C ${remoteDir} update-ref -d ${exportRef} >/dev/null 2>&1 || true`,
];
return [
"set -e",
input.cleanupBundle ? `cleanup() { ${cleanupParts.join("; ")}; }` : "",
input.cleanupBundle ? "trap cleanup EXIT" : "",
`mkdir -p ${shellQuote(path.posix.dirname(input.bundlePath))}`,
`rm -f ${bundlePath}`,
`git -C ${remoteDir} cat-file -e ${baseSha}^{commit}`,
`commit_count=$(git -C ${remoteDir} rev-list --count HEAD --not ${baseSha})`,
'if [ "$commit_count" -gt 0 ]; then',
` git -C ${remoteDir} update-ref ${exportRef} HEAD`,
` git -C ${remoteDir} bundle create ${bundlePath} ${exportRef} --not ${baseSha} >/dev/null`,
"else",
` : > ${bundlePath}`,
"fi",
input.catBundle ? `cat ${bundlePath}` : "",
].filter(Boolean).join("\n");
}
export async function integrateImportedGitHead(input: {
localDir: string;
importedHead: string;
}): Promise<void> {
const isConcurrentRefUpdateError = (error: unknown) => {
const message = error instanceof Error ? error.message : String(error);
return message.includes("cannot lock ref") && message.includes("expected");
};
for (let attempt = 0; attempt < 5; attempt += 1) {
const snapshot = await readGitWorkspaceSnapshot(input.localDir);
if (!snapshot) return;
const currentHead = snapshot.headCommit;
if (!currentHead || currentHead === input.importedHead) return;
const headRef = snapshot.branchName ? `refs/heads/${snapshot.branchName}` : "HEAD";
const mergeBase = await runLocalGit(input.localDir, ["merge-base", currentHead, input.importedHead], {
timeout: 10_000,
maxBuffer: 16 * 1024,
}).catch(() => null);
const mergeBaseHead = mergeBase?.stdout.trim() ?? "";
if (mergeBaseHead === input.importedHead) {
return;
}
if (mergeBaseHead === currentHead) {
try {
await runLocalGit(input.localDir, ["update-ref", headRef, input.importedHead, currentHead], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
return;
} catch (error) {
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
throw error;
}
}
let mergedTree;
try {
mergedTree = await runLocalGit(input.localDir, ["merge-tree", "--write-tree", currentHead, input.importedHead], {
timeout: 60_000,
maxBuffer: 256 * 1024,
});
} catch (error) {
const reason = error instanceof Error ? error.message : String(error);
throw new Error(
`Failed to merge concurrent remote git histories for ${currentHead.slice(0, 12)} and ${input.importedHead.slice(0, 12)}: ${reason}`,
);
}
const mergedTreeId = mergedTree.stdout.trim().split("\n")[0]?.trim() ?? "";
if (!mergedTreeId) {
throw new Error("Failed to compute a merged git tree for workspace restore.");
}
const mergeCommit = await runLocalGit(
input.localDir,
[
"commit-tree",
mergedTreeId,
"-p",
currentHead,
"-p",
input.importedHead,
"-m",
`Paperclip remote git sync merge ${input.importedHead.slice(0, 12)}`,
],
{
timeout: 60_000,
maxBuffer: 64 * 1024,
},
);
try {
await runLocalGit(input.localDir, ["update-ref", headRef, mergeCommit.stdout.trim(), currentHead], {
timeout: 10_000,
maxBuffer: 16 * 1024,
});
return;
} catch (error) {
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
throw error;
}
}
throw new Error(`Failed to integrate concurrent remote git history for ${input.importedHead.slice(0, 12)} after multiple retries.`);
}
@@ -1,4 +1,5 @@
import path from "node:path";
import { GIT_ARCHIVE_EXCLUDES } from "./git-workspace-sync.js";
import {
type SshRemoteExecutionSpec,
prepareWorkspaceForSshExecution,
@@ -90,7 +91,7 @@ export async function prepareRemoteManagedRuntime(input: {
remoteDir: workspaceRemoteDir,
onProgress: input.onProgress,
});
const restoreExclude = preparedWorkspace.gitBacked ? [".git", ".paperclip-runtime"] : [".paperclip-runtime"];
const restoreExclude = preparedWorkspace.gitBacked ? [...GIT_ARCHIVE_EXCLUDES, ".paperclip-runtime"] : [".paperclip-runtime"];
const baselineSnapshot = await captureDirectorySnapshot(input.workspaceLocalDir, {
exclude: restoreExclude,
});
@@ -13,6 +13,20 @@ import {
const execFile = promisify(execFileCallback);
async function git(cwd: string, args: string[]): Promise<string> {
const { stdout } = await execFile("git", ["-C", cwd, ...args], {
maxBuffer: 32 * 1024 * 1024,
});
return stdout.trim();
}
async function listTarMembers(rootDir: string, name: string, bytes: Buffer): Promise<string[]> {
const tarPath = path.join(rootDir, name);
await writeFile(tarPath, bytes);
const { stdout } = await execFile("tar", ["-tf", tarPath], { maxBuffer: 32 * 1024 * 1024 });
return stdout.split("\n").map((line) => line.trim()).filter(Boolean);
}
describe("sandbox managed runtime", () => {
const cleanupDirs: string[] = [];
@@ -131,6 +145,224 @@ describe("sandbox managed runtime", () => {
await expect(readFile(path.join(localWorkspaceDir, ".paperclip-runtime", "state.json"), "utf8")).resolves.toBe("{}\n");
});
it("syncs git-backed workspaces through a shallow standalone clone and keeps .git out of archives", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-git-"));
cleanupDirs.push(rootDir);
const sourceRepoDir = path.join(rootDir, "source-repo");
const localWorkspaceDir = path.join(rootDir, "local-worktree");
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
await mkdir(sourceRepoDir, { recursive: true });
await git(sourceRepoDir, ["init"]);
await git(sourceRepoDir, ["checkout", "-b", "main"]);
await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]);
await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(sourceRepoDir, ".gitignore"), "node_modules/\n", "utf8");
await writeFile(path.join(sourceRepoDir, "tracked.txt"), "base\n", "utf8");
await writeFile(path.join(sourceRepoDir, "clean.txt"), "from git\n", "utf8");
await writeFile(path.join(sourceRepoDir, "deleted.txt"), "delete me\n", "utf8");
await git(sourceRepoDir, ["add", ".gitignore", "tracked.txt", "clean.txt", "deleted.txt"]);
await git(sourceRepoDir, ["commit", "-m", "base"]);
await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]);
expect((await lstat(path.join(localWorkspaceDir, ".git"))).isFile()).toBe(true);
await mkdir(path.join(localWorkspaceDir, "node_modules"), { recursive: true });
await writeFile(path.join(localWorkspaceDir, "tracked.txt"), "dirty local\n", "utf8");
await writeFile(path.join(localWorkspaceDir, "untracked.txt"), "from local\n", "utf8");
await writeFile(path.join(localWorkspaceDir, "node_modules", "cache.bin"), "do not upload\n", "utf8");
await rm(path.join(localWorkspaceDir, "deleted.txt"));
const uploadedTars: { remotePath: string; bytes: Buffer }[] = [];
const downloadedTars: { remotePath: string; bytes: Buffer }[] = [];
const client: SandboxManagedRuntimeClient = {
makeDir: async (remotePath) => {
await mkdir(remotePath, { recursive: true });
},
writeFile: async (remotePath, bytes) => {
await mkdir(path.dirname(remotePath), { recursive: true });
const buffer = Buffer.from(bytes);
if (remotePath.endsWith("-upload.tar")) uploadedTars.push({ remotePath, bytes: buffer });
await writeFile(remotePath, buffer);
},
readFile: async (remotePath) => {
const buffer = await readFile(remotePath);
if (remotePath.endsWith("workspace-download.tar")) downloadedTars.push({ remotePath, bytes: buffer });
return buffer;
},
listFiles: async () => [],
remove: async (remotePath) => {
await rm(remotePath, { recursive: true, force: true });
},
run: async (command) => {
await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 });
},
};
const prepared = await prepareSandboxManagedRuntime({
spec: {
transport: "sandbox",
provider: "test",
sandboxId: "sandbox-1",
remoteCwd: remoteWorkspaceDir,
timeoutMs: 30_000,
apiKey: null,
},
adapterKey: "test-adapter",
client,
workspaceLocalDir: localWorkspaceDir,
});
expect((await lstat(path.join(remoteWorkspaceDir, ".git"))).isDirectory()).toBe(true);
await expect(readFile(path.join(remoteWorkspaceDir, ".git", "shallow"), "utf8")).resolves.toContain(
await git(localWorkspaceDir, ["rev-parse", "HEAD"]),
);
expect(await git(remoteWorkspaceDir, ["rev-list", "--count", "HEAD"])).toBe("1");
expect(await git(remoteWorkspaceDir, ["status", "--short"])).toContain("M tracked.txt");
expect(await git(remoteWorkspaceDir, ["status", "--short"])).toContain("?? untracked.txt");
await expect(readFile(path.join(remoteWorkspaceDir, "deleted.txt"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
const gitUpload = uploadedTars.find((entry) => path.posix.basename(entry.remotePath) === "git-workspace-upload.tar");
const workspaceUpload = uploadedTars.find((entry) => path.posix.basename(entry.remotePath) === "workspace-upload.tar");
expect(gitUpload).toBeDefined();
expect(workspaceUpload).toBeDefined();
const gitMembers = await listTarMembers(rootDir, "git-upload-list.tar", gitUpload!.bytes);
const workspaceMembers = await listTarMembers(rootDir, "workspace-upload-list.tar", workspaceUpload!.bytes);
expect(gitMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(true);
expect(workspaceMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(false);
expect(workspaceMembers).toContain("tracked.txt");
expect(workspaceMembers).toContain("untracked.txt");
expect(workspaceMembers).not.toContain("clean.txt");
expect(workspaceMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
await git(remoteWorkspaceDir, ["config", "user.name", "Paperclip Sandbox"]);
await git(remoteWorkspaceDir, ["config", "user.email", "sandbox@paperclip.dev"]);
await git(remoteWorkspaceDir, ["add", "-A"]);
await git(remoteWorkspaceDir, ["commit", "-m", "sandbox update"]);
await writeFile(path.join(remoteWorkspaceDir, "tracked.txt"), "remote dirty\n", "utf8");
await writeFile(path.join(remoteWorkspaceDir, "remote-only.txt"), "from sandbox\n", "utf8");
await prepared.restoreWorkspace();
expect((await lstat(path.join(localWorkspaceDir, ".git"))).isFile()).toBe(true);
expect(await git(localWorkspaceDir, ["log", "-1", "--pretty=%s"])).toBe("sandbox update");
await expect(readFile(path.join(localWorkspaceDir, "tracked.txt"), "utf8")).resolves.toBe("remote dirty\n");
await expect(readFile(path.join(localWorkspaceDir, "remote-only.txt"), "utf8")).resolves.toBe("from sandbox\n");
await expect(readFile(path.join(localWorkspaceDir, "deleted.txt"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
await expect(readFile(path.join(localWorkspaceDir, "node_modules", "cache.bin"), "utf8")).resolves.toBe("do not upload\n");
expect(downloadedTars).toHaveLength(1);
const downloadMembers = await listTarMembers(rootDir, "workspace-download-list.tar", downloadedTars[0]!.bytes);
expect(downloadMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(false);
expect(downloadMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
});
it("excludes unignored dependency trees from git-backed workspace overlay archives", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-unignored-deps-"));
cleanupDirs.push(rootDir);
const sourceRepoDir = path.join(rootDir, "source-repo");
const localWorkspaceDir = path.join(rootDir, "local-worktree");
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
await mkdir(sourceRepoDir, { recursive: true });
await git(sourceRepoDir, ["init"]);
await git(sourceRepoDir, ["checkout", "-b", "main"]);
await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]);
await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]);
await mkdir(path.join(sourceRepoDir, "src"), { recursive: true });
await writeFile(path.join(sourceRepoDir, "src", "tracked.ts"), "export const tracked = true;\n", "utf8");
await git(sourceRepoDir, ["add", "src/tracked.ts"]);
await git(sourceRepoDir, ["commit", "-m", "base"]);
await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]);
await mkdir(path.join(localWorkspaceDir, "node_modules", "root-package"), { recursive: true });
await mkdir(path.join(localWorkspaceDir, "packages", "ui", "node_modules", "nested-package"), { recursive: true });
await writeFile(path.join(localWorkspaceDir, "node_modules", "root-package", "cache.bin"), "root dependency\n", "utf8");
await writeFile(
path.join(localWorkspaceDir, "packages", "ui", "node_modules", "nested-package", "cache.bin"),
"nested dependency\n",
"utf8",
);
await writeFile(path.join(localWorkspaceDir, "src", "local-only.ts"), "export const local = true;\n", "utf8");
const uploadedTars: { remotePath: string; bytes: Buffer }[] = [];
const downloadedTars: { remotePath: string; bytes: Buffer }[] = [];
const client: SandboxManagedRuntimeClient = {
makeDir: async (remotePath) => {
await mkdir(remotePath, { recursive: true });
},
writeFile: async (remotePath, bytes) => {
await mkdir(path.dirname(remotePath), { recursive: true });
const buffer = Buffer.from(bytes);
if (remotePath.endsWith("-upload.tar")) uploadedTars.push({ remotePath, bytes: buffer });
await writeFile(remotePath, buffer);
},
readFile: async (remotePath) => {
const buffer = await readFile(remotePath);
if (remotePath.endsWith("workspace-download.tar")) downloadedTars.push({ remotePath, bytes: buffer });
return buffer;
},
listFiles: async () => [],
remove: async (remotePath) => {
await rm(remotePath, { recursive: true, force: true });
},
run: async (command) => {
await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 });
},
};
const prepared = await prepareSandboxManagedRuntime({
spec: {
transport: "sandbox",
provider: "test",
sandboxId: "sandbox-1",
remoteCwd: remoteWorkspaceDir,
timeoutMs: 30_000,
apiKey: null,
},
adapterKey: "test-adapter",
client,
workspaceLocalDir: localWorkspaceDir,
});
const workspaceUpload = uploadedTars.find((entry) => path.posix.basename(entry.remotePath) === "workspace-upload.tar");
expect(workspaceUpload).toBeDefined();
const workspaceMembers = await listTarMembers(rootDir, "unignored-deps-workspace-upload.tar", workspaceUpload!.bytes);
expect(workspaceMembers).toContain("src/local-only.ts");
expect(workspaceMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
expect(workspaceMembers.some((entry) => entry.includes("/node_modules/") || entry.endsWith("/node_modules"))).toBe(false);
await expect(readFile(path.join(remoteWorkspaceDir, "src", "local-only.ts"), "utf8")).resolves.toBe("export const local = true;\n");
await expect(readFile(path.join(remoteWorkspaceDir, "node_modules", "root-package", "cache.bin"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
await expect(
readFile(path.join(remoteWorkspaceDir, "packages", "ui", "node_modules", "nested-package", "cache.bin"), "utf8"),
).rejects.toMatchObject({ code: "ENOENT" });
await mkdir(path.join(remoteWorkspaceDir, "node_modules", "sandbox-package"), { recursive: true });
await mkdir(path.join(remoteWorkspaceDir, "packages", "ui", "node_modules", "sandbox-package"), { recursive: true });
await writeFile(path.join(remoteWorkspaceDir, "node_modules", "sandbox-package", "cache.bin"), "sandbox root dependency\n", "utf8");
await writeFile(
path.join(remoteWorkspaceDir, "packages", "ui", "node_modules", "sandbox-package", "cache.bin"),
"sandbox nested dependency\n",
"utf8",
);
await writeFile(path.join(remoteWorkspaceDir, "src", "remote-only.ts"), "export const remote = true;\n", "utf8");
await prepared.restoreWorkspace();
await expect(readFile(path.join(localWorkspaceDir, "node_modules", "root-package", "cache.bin"), "utf8")).resolves.toBe("root dependency\n");
await expect(
readFile(path.join(localWorkspaceDir, "packages", "ui", "node_modules", "nested-package", "cache.bin"), "utf8"),
).resolves.toBe("nested dependency\n");
await expect(readFile(path.join(localWorkspaceDir, "src", "remote-only.ts"), "utf8")).resolves.toBe("export const remote = true;\n");
await expect(readFile(path.join(localWorkspaceDir, "node_modules", "sandbox-package", "cache.bin"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
expect(downloadedTars).toHaveLength(1);
const downloadMembers = await listTarMembers(rootDir, "unignored-deps-workspace-download.tar", downloadedTars[0]!.bytes);
expect(downloadMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(false);
expect(downloadMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
expect(downloadMembers.some((entry) => entry.includes("/node_modules/") || entry.endsWith("/node_modules"))).toBe(false);
});
it("builds workspace/asset tarballs without a './' self-entry (so untar does not chmod/utime an unowned target dir)", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-tarself-"));
cleanupDirs.push(rootDir);
@@ -3,10 +3,44 @@ import { constants as fsConstants, promises as fs } from "node:fs";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import {
buildRemoteGitDeltaBundleScript,
createImportedGitRef,
createRemoteGitExportRef,
deleteLocalGitRef,
fetchGitBundleIntoLocalRef,
GIT_ARCHIVE_EXCLUDES,
integrateImportedGitHead,
readGitWorkspaceSnapshot,
withShallowGitWorkspaceClone,
} from "./git-workspace-sync.js";
import { captureDirectorySnapshot, mergeDirectoryWithBaseline } from "./workspace-restore-merge.js";
import { createRuntimeProgressReporter, type RuntimeProgressSink } from "./runtime-progress.js";
import {
createRuntimeProgressReporter,
type RuntimeProgressDirection,
type RuntimeProgressPhase,
type RuntimeProgressSink,
} from "./runtime-progress.js";
import { isRelativePathOrDescendant, shouldExcludePath } from "./exclude-patterns.js";
const execFile = promisify(execFileCallback);
const SANDBOX_WORKSPACE_HEAVY_DIR_NAMES = [
"node_modules",
"vendor",
"dist",
"build",
"out",
"coverage",
".next",
".turbo",
".cache",
] as const;
const SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES = SANDBOX_WORKSPACE_HEAVY_DIR_NAMES.flatMap((entry) => [
entry,
`${entry}/*`,
`*/${entry}`,
`*/${entry}/*`,
]);
export interface SandboxRemoteExecutionSpec {
transport: "sandbox";
@@ -208,8 +242,28 @@ async function walkDirectory(root: string, relative = ""): Promise<string[]> {
return out.sort((left, right) => right.length - left.length);
}
function isRelativePathOrDescendant(relative: string, candidate: string): boolean {
return relative === candidate || relative.startsWith(`${candidate}/`);
async function copyWorkspaceEntry(sourceRoot: string, targetRoot: string, relative: string): Promise<void> {
const sourcePath = path.join(sourceRoot, relative);
const targetPath = path.join(targetRoot, relative);
const stats = await fs.lstat(sourcePath);
if (stats.isDirectory()) {
await fs.mkdir(targetPath, { recursive: true });
return;
}
await fs.mkdir(path.dirname(targetPath), { recursive: true });
await fs.rm(targetPath, { recursive: true, force: true }).catch(() => undefined);
if (stats.isSymbolicLink()) {
const linkTarget = await fs.readlink(sourcePath);
await fs.symlink(linkTarget, targetPath);
return;
}
await fs.copyFile(sourcePath, targetPath, fsConstants.COPYFILE_FICLONE).catch(async () => {
await fs.copyFile(sourcePath, targetPath);
});
await fs.chmod(targetPath, stats.mode);
}
export async function mirrorDirectory(
@@ -231,33 +285,24 @@ export async function mirrorDirectory(
}
}
const copyEntry = async (relative: string) => {
const sourcePath = path.join(sourceDir, relative);
const targetPath = path.join(targetDir, relative);
const stats = await fs.lstat(sourcePath);
if (stats.isDirectory()) {
await fs.mkdir(targetPath, { recursive: true });
return;
}
await fs.mkdir(path.dirname(targetPath), { recursive: true });
await fs.rm(targetPath, { recursive: true, force: true }).catch(() => undefined);
if (stats.isSymbolicLink()) {
const linkTarget = await fs.readlink(sourcePath);
await fs.symlink(linkTarget, targetPath);
return;
}
await fs.copyFile(sourcePath, targetPath, fsConstants.COPYFILE_FICLONE).catch(async () => {
await fs.copyFile(sourcePath, targetPath);
});
await fs.chmod(targetPath, stats.mode);
};
const entries = (await walkDirectory(sourceDir)).sort((left, right) => left.localeCompare(right));
for (const relative of entries) {
await copyEntry(relative);
await copyWorkspaceEntry(sourceDir, targetDir, relative);
}
}
async function copySelectedWorkspaceEntries(input: {
sourceDir: string;
targetDir: string;
relativePaths: string[];
exclude: string[];
}): Promise<void> {
await fs.mkdir(input.targetDir, { recursive: true });
for (const relative of input.relativePaths) {
if (shouldExcludePath(relative, input.exclude)) continue;
const sourceStats = await fs.lstat(path.join(input.sourceDir, relative)).catch(() => null);
if (!sourceStats) continue;
await copyWorkspaceEntry(input.sourceDir, input.targetDir, relative);
}
}
@@ -275,15 +320,37 @@ function tarExcludeFlags(exclude: string[] | undefined): string {
return ["._*", ...(exclude ?? [])].map((entry) => `--exclude ${shellQuote(entry)}`).join(" ");
}
function mergeExcludes(...groups: Array<string[] | undefined>): string[] {
return [...new Set(groups.flatMap((group) => group ?? []))];
}
function preserveFindArgs(entries: string[]): string {
return entries.map((entry) => `! -name ${shellQuote(entry)}`).join(" ");
}
async function removeDeletedPathsInSandbox(input: {
client: SandboxManagedRuntimeClient;
spec: SandboxRemoteExecutionSpec;
remoteDir: string;
deletedPaths: string[];
}): Promise<void> {
if (input.deletedPaths.length === 0) return;
const quotedPaths = input.deletedPaths.map((entry) => shellQuote(entry)).join(" ");
await input.client.run(
`sh -c ${shellQuote(`cd ${shellQuote(input.remoteDir)} && rm -rf -- ${quotedPaths}`)}`,
{ timeoutMs: input.spec.timeoutMs },
);
}
// Bridge a single byte-level transfer to the throttled progress reporter. The
// transport reports decoded bytes via `options.onProgress`; the reporter turns
// them into a throttled, fully-formatted log line. `finish()` emits the terminal
// completion line (idempotent) once the transfer returns.
function makeTransferProgress(
sink: RuntimeProgressSink | undefined,
phase: "Syncing" | "Restoring",
direction: "to" | "from",
label: string,
phase: RuntimeProgressPhase,
direction: RuntimeProgressDirection,
label?: string,
): { options: SandboxTransferProgressOptions | undefined; finish: () => Promise<void> } {
if (!sink) return { options: undefined, finish: async () => {} };
const reporter = createRuntimeProgressReporter({
@@ -320,16 +387,75 @@ export async function prepareSandboxManagedRuntime(input: {
}): Promise<PreparedSandboxManagedRuntime> {
const workspaceRemoteDir = input.workspaceRemoteDir ?? input.spec.remoteCwd;
const runtimeRootDir = path.posix.join(workspaceRemoteDir, ".paperclip-runtime", input.adapterKey);
const gitSnapshot = await readGitWorkspaceSnapshot(input.workspaceLocalDir);
const gitIgnoredExcludes = gitSnapshot?.ignoredPaths;
const workspaceArchiveExclude = mergeExcludes(
SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES,
[...GIT_ARCHIVE_EXCLUDES],
input.workspaceExclude,
gitIgnoredExcludes,
);
const restoreExclude = mergeExcludes(
SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES,
[...GIT_ARCHIVE_EXCLUDES],
[".paperclip-runtime"],
input.preserveAbsentOnRestore,
input.workspaceExclude,
gitIgnoredExcludes,
);
const baselineSnapshot = await captureDirectorySnapshot(input.workspaceLocalDir, {
exclude: [...new Set([".paperclip-runtime", ...(input.preserveAbsentOnRestore ?? []), ...(input.workspaceExclude ?? [])])],
exclude: restoreExclude,
});
await withTempDir("paperclip-sandbox-sync-", async (tempDir) => {
const preservedNames = new Set([
".paperclip-runtime",
...(gitSnapshot ? [".git"] : []),
...(input.preserveAbsentOnRestore ?? []),
]);
if (gitSnapshot) {
await withShallowGitWorkspaceClone({
localDir: input.workspaceLocalDir,
snapshot: gitSnapshot,
}, async (cloneDir) => {
const gitTarPath = path.join(tempDir, "git-workspace.tar");
await createTarballFromDirectory({
localDir: cloneDir,
archivePath: gitTarPath,
exclude: [".paperclip-runtime"],
});
const gitTarBytes = await fs.readFile(gitTarPath);
const remoteGitTar = path.posix.join(runtimeRootDir, "git-workspace-upload.tar");
await input.client.makeDir(runtimeRootDir);
const gitUpload = makeTransferProgress(input.onProgress, "Syncing", "to", "git history");
await input.client.writeFile(remoteGitTar, toArrayBuffer(gitTarBytes), gitUpload.options);
await gitUpload.finish();
await input.client.run(
`sh -c ${shellQuote(
`mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${preserveFindArgs([".paperclip-runtime"])} -exec rm -rf -- {} + && ` +
`tar -xf ${shellQuote(remoteGitTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
`rm -f ${shellQuote(remoteGitTar)}`,
)}`,
{ timeoutMs: input.spec.timeoutMs },
);
});
}
const workspaceTarPath = path.join(tempDir, "workspace.tar");
const workspaceArchiveDir = gitSnapshot ? path.join(tempDir, "workspace-overlay") : input.workspaceLocalDir;
if (gitSnapshot) {
await copySelectedWorkspaceEntries({
sourceDir: input.workspaceLocalDir,
targetDir: workspaceArchiveDir,
relativePaths: gitSnapshot.overlayPaths,
exclude: workspaceArchiveExclude,
});
}
await createTarballFromDirectory({
localDir: input.workspaceLocalDir,
localDir: workspaceArchiveDir,
archivePath: workspaceTarPath,
exclude: input.workspaceExclude,
exclude: gitSnapshot ? undefined : workspaceArchiveExclude,
});
const workspaceTarBytes = await fs.readFile(workspaceTarPath);
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-upload.tar");
@@ -341,17 +467,26 @@ export async function prepareSandboxManagedRuntime(input: {
workspaceUpload.options,
);
await workspaceUpload.finish();
const preservedNames = new Set([".paperclip-runtime", ...(input.preserveAbsentOnRestore ?? [])]);
const findPreserveArgs = [...preservedNames].map((entry) => `! -name ${shellQuote(entry)}`).join(" ");
const extractWorkspaceTarCommand = gitSnapshot
? `mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
`rm -f ${shellQuote(remoteWorkspaceTar)}`
: `mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${preserveFindArgs([...preservedNames])} -exec rm -rf -- {} + && ` +
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
`rm -f ${shellQuote(remoteWorkspaceTar)}`;
await input.client.run(
`sh -c ${shellQuote(
`mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${findPreserveArgs} -exec rm -rf -- {} + && ` +
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
`rm -f ${shellQuote(remoteWorkspaceTar)}`,
)}`,
`sh -c ${shellQuote(extractWorkspaceTarCommand)}`,
{ timeoutMs: input.spec.timeoutMs },
);
if (gitSnapshot) {
await removeDeletedPathsInSandbox({
client: input.client,
spec: input.spec,
remoteDir: workspaceRemoteDir,
deletedPaths: gitSnapshot.deletedPaths,
});
}
for (const asset of input.assets ?? []) {
const assetTarPath = path.join(tempDir, `${asset.key}.tar`);
@@ -392,31 +527,76 @@ export async function prepareSandboxManagedRuntime(input: {
restoreWorkspace: async (onProgress?: RuntimeProgressSink) => {
const restoreSink = onProgress ?? input.onProgress;
await withTempDir("paperclip-sandbox-restore-", async (tempDir) => {
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-download.tar");
await input.client.run(
`sh -c ${shellQuote(
`mkdir -p ${shellQuote(runtimeRootDir)} && ` +
`tar -cf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} ` +
`${tarExcludeFlags(input.workspaceExclude)} .`,
)}`,
{ timeoutMs: input.spec.timeoutMs },
);
const workspaceRestore = makeTransferProgress(restoreSink, "Restoring", "from", "workspace");
const archiveBytes = await input.client.readFile(remoteWorkspaceTar, workspaceRestore.options);
await workspaceRestore.finish();
await input.client.remove(remoteWorkspaceTar).catch(() => undefined);
const localArchivePath = path.join(tempDir, "workspace.tar");
const extractedDir = path.join(tempDir, "workspace");
await fs.writeFile(localArchivePath, toBuffer(archiveBytes));
await extractTarballToDirectory({
archivePath: localArchivePath,
localDir: extractedDir,
});
await mergeDirectoryWithBaseline({
baseline: baselineSnapshot,
sourceDir: extractedDir,
targetDir: input.workspaceLocalDir,
});
let importedRef: string | null = null;
let importedHead: string | null = null;
try {
if (gitSnapshot) {
importedRef = createImportedGitRef("sandbox");
const remoteGitBundle = path.posix.join(runtimeRootDir, "git-delta.bundle");
const exportRef = createRemoteGitExportRef("sandbox");
await input.client.run(
`sh -c ${shellQuote(buildRemoteGitDeltaBundleScript({
remoteDir: workspaceRemoteDir,
baseSha: gitSnapshot.headCommit,
exportRef,
bundlePath: remoteGitBundle,
}))}`,
{ timeoutMs: input.spec.timeoutMs },
);
const gitExport = makeTransferProgress(restoreSink, "Exporting git history", "from");
const bundleBytes = await input.client.readFile(remoteGitBundle, gitExport.options);
await gitExport.finish();
await input.client.remove(remoteGitBundle).catch(() => undefined);
const bundlePath = path.join(tempDir, "git-delta.bundle");
await fs.writeFile(bundlePath, toBuffer(bundleBytes));
importedHead = await fetchGitBundleIntoLocalRef({
localDir: input.workspaceLocalDir,
bundlePath,
exportRef,
importedRef,
baseSha: gitSnapshot.headCommit,
});
}
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-download.tar");
await input.client.run(
`sh -c ${shellQuote(
`mkdir -p ${shellQuote(runtimeRootDir)} && ` +
`tar -cf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} ` +
`${tarExcludeFlags(restoreExclude)} .`,
)}`,
{ timeoutMs: input.spec.timeoutMs },
);
const workspaceRestore = makeTransferProgress(restoreSink, "Restoring", "from", "workspace");
const archiveBytes = await input.client.readFile(remoteWorkspaceTar, workspaceRestore.options);
await workspaceRestore.finish();
await input.client.remove(remoteWorkspaceTar).catch(() => undefined);
const localArchivePath = path.join(tempDir, "workspace.tar");
const extractedDir = path.join(tempDir, "workspace");
await fs.writeFile(localArchivePath, toBuffer(archiveBytes));
await extractTarballToDirectory({
archivePath: localArchivePath,
localDir: extractedDir,
});
const gitHeadToIntegrate = importedHead;
await mergeDirectoryWithBaseline({
baseline: baselineSnapshot,
sourceDir: extractedDir,
targetDir: input.workspaceLocalDir,
beforeApply: gitHeadToIntegrate
? async () => {
await integrateImportedGitHead({
localDir: input.workspaceLocalDir,
importedHead: gitHeadToIntegrate,
});
}
: undefined,
});
} finally {
if (importedRef) {
await deleteLocalGitRef({ localDir: input.workspaceLocalDir, ref: importedRef });
}
}
});
},
};
@@ -2,6 +2,7 @@ import { createHash } from "node:crypto";
import { createReadStream } from "node:fs";
import { constants as fsConstants, promises as fs } from "node:fs";
import path from "node:path";
import { shouldExcludePath } from "./exclude-patterns.js";
type SnapshotEntry =
| { kind: "dir" }
@@ -13,14 +14,6 @@ export interface DirectorySnapshot {
entries: Map<string, SnapshotEntry>;
}
function isRelativePathOrDescendant(relative: string, candidate: string): boolean {
return relative === candidate || relative.startsWith(`${candidate}/`);
}
function shouldExclude(relative: string, exclude: readonly string[]): boolean {
return exclude.some((candidate) => isRelativePathOrDescendant(relative, candidate));
}
async function hashFile(filePath: string): Promise<string> {
return await new Promise((resolve, reject) => {
const hash = createHash("sha256");
@@ -43,7 +36,7 @@ async function walkDirectory(
for (const entry of entries) {
const nextRelative = relative ? path.posix.join(relative, entry.name) : entry.name;
if (shouldExclude(nextRelative, exclude)) continue;
if (shouldExcludePath(nextRelative, exclude)) continue;
const fullPath = path.join(root, nextRelative);
const stats = await fs.lstat(fullPath);
@@ -313,6 +313,82 @@ describe("resolveExecutionRunAdapterConfig", () => {
details: { code: "low_trust_inline_sensitive_env_denied" },
});
});
it("fails push-capability preflight when no GitHub write credential is bound at agent or project scope", async () => {
await expect(resolveExecutionRunAdapterConfig({
companyId: "company-1",
agentId: "agent-1",
issueId: "issue-1",
executionRunConfig: { env: { AGENT_ONLY: "agent-only" } },
projectEnv: { PROJECT_ONLY: "project-only" },
requiredScopedEnvBinding: {
keys: ["GH_TOKEN", "GITHUB_TOKEN"],
consumerScopes: ["agent", "project"],
reason: "push_write_credential_missing",
remediation: "GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN bound at project or agent scope.",
},
secretsSvc: {
resolveAdapterConfigForRuntime: vi.fn(),
resolveEnvBindings: vi.fn(),
} as any,
})).rejects.toMatchObject({
code: "configuration_incomplete",
message: expect.stringContaining("GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN"),
resultJson: {
configurationIncomplete: {
reason: "push_write_credential_missing",
requiredEnvKeys: ["GH_TOKEN", "GITHUB_TOKEN"],
requiredScopes: ["agent", "project"],
missingBindings: [],
},
},
});
});
it("passes push-capability preflight when a project-scoped GitHub credential is configured", async () => {
const resolveAdapterConfigForRuntime = vi.fn().mockResolvedValue({
config: { env: { AGENT_ONLY: "agent-only" } },
secretKeys: new Set<string>(),
manifest: [],
});
const resolveEnvBindings = vi.fn().mockResolvedValue({
env: { GH_TOKEN: "github-token" },
secretKeys: new Set(["GH_TOKEN"]),
manifest: [],
});
const collectMissingRuntimeBindings = vi.fn().mockResolvedValue([]);
const result = await resolveExecutionRunAdapterConfig({
companyId: "company-1",
agentId: "agent-1",
issueId: "issue-1",
projectId: "project-1",
executionRunConfig: { env: { AGENT_ONLY: "agent-only" } },
projectEnv: { GH_TOKEN: { type: "plain", value: "github-token" } },
requiredScopedEnvBinding: {
keys: ["GH_TOKEN", "GITHUB_TOKEN"],
consumerScopes: ["agent", "project"],
reason: "push_write_credential_missing",
remediation: "GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN bound at project or agent scope.",
},
secretsSvc: {
resolveAdapterConfigForRuntime,
resolveEnvBindings,
collectMissingRuntimeBindings,
} as any,
});
expect(result.resolvedConfig.env).toEqual({
AGENT_ONLY: "agent-only",
GH_TOKEN: "github-token",
});
expect(resolveEnvBindings).toHaveBeenCalledOnce();
expect(collectMissingRuntimeBindings).toHaveBeenCalledTimes(2);
expect(collectMissingRuntimeBindings.mock.calls[1]?.[2]).toMatchObject({
consumerType: "project",
consumerId: "project-1",
});
});
});
describe("extractMentionedSkillIdsFromSources", () => {
@@ -1,3 +1,8 @@
import { execFile as execFileCallback } from "node:child_process";
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { promisify } from "node:util";
import { describe, expect, it } from "vitest";
import type { agents } from "@paperclipai/db";
import { sessionCodec as codexSessionCodec } from "@paperclipai/adapter-codex-local/server";
@@ -5,6 +10,7 @@ import { resolveDefaultAgentWorkspaceDir } from "../home-paths.js";
import {
applyPersistedExecutionWorkspaceConfig,
assertGitSensitiveAdapterWorkspaceValid,
assertPushCapabilityCheckoutValid,
buildRealizedExecutionWorkspaceFromPersisted,
buildExplicitResumeSessionOverride,
deriveTaskKeyWithHeartbeatFallback,
@@ -16,6 +22,7 @@ import {
prioritizeProjectWorkspaceCandidatesForRun,
parseSessionCompactionPolicy,
resolveNextSessionState,
requiresPushCapabilityPreflight,
resolveWorkspaceAfterLowTrustPreflight,
resolveRuntimeSessionParamsForWorkspace,
shouldDeferFollowupWakeForSameIssue,
@@ -29,6 +36,8 @@ import {
} from "../services/heartbeat.ts";
import type { TrustPresetResolution } from "../services/trust-preset-resolver.ts";
const execFile = promisify(execFileCallback);
function buildResolvedWorkspace(overrides: Partial<ResolvedWorkspaceForRun> = {}): ResolvedWorkspaceForRun {
return {
cwd: "/tmp/project",
@@ -105,6 +114,19 @@ function buildWorkspaceValidationInput(
};
}
async function runGit(cwd: string, args: string[]) {
await execFile("git", args, { cwd });
}
async function createGitCheckout(options: { withRemote: boolean }) {
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-push-preflight-"));
await runGit(root, ["init"]);
if (options.withRemote) {
await runGit(root, ["remote", "add", "origin", "https://github.com/example/repo.git"]);
}
return root;
}
async function expectWorkspaceValidationFailure(
input: WorkspaceValidationInput,
reason: string,
@@ -377,6 +399,72 @@ describe("assertGitSensitiveAdapterWorkspaceValid", () => {
});
});
describe("assertPushCapabilityCheckoutValid", () => {
it("rejects a GitHub PR workflow checkout without a configured push remote", async () => {
const cwd = await createGitCheckout({ withRemote: false });
try {
await expect(assertPushCapabilityCheckoutValid({
enabled: true,
issue: {
id: "issue-1",
identifier: "PAP-1",
},
cwd,
})).rejects.toMatchObject({
code: "workspace_validation_failed",
message: expect.stringContaining("has no configured push remote"),
resultJson: {
workspaceValidation: expect.objectContaining({
reason: "missing_git_push_remote",
issueId: "issue-1",
executionWorkspaceCwd: cwd,
}),
},
});
} finally {
await fs.rm(cwd, { recursive: true, force: true });
}
});
it("allows a GitHub PR workflow checkout when a push remote is configured", async () => {
const cwd = await createGitCheckout({ withRemote: true });
try {
await expect(assertPushCapabilityCheckoutValid({
enabled: true,
issue: {
id: "issue-1",
identifier: "PAP-1",
},
cwd,
})).resolves.toBeUndefined();
} finally {
await fs.rm(cwd, { recursive: true, force: true });
}
});
});
describe("requiresPushCapabilityPreflight", () => {
it("only enables the guard when the issue explicitly mentions the GitHub PR workflow skill", () => {
expect(requiresPushCapabilityPreflight({
adapterType: "codex_local",
issueId: "issue-1",
explicitRunScopedSkillKeys: ["paperclipai/bundled/software-development/github-pr-workflow"],
})).toBe(true);
expect(requiresPushCapabilityPreflight({
adapterType: "codex_local",
issueId: "issue-1",
explicitRunScopedSkillKeys: [],
})).toBe(false);
expect(requiresPushCapabilityPreflight({
adapterType: "cursor-cloud",
issueId: "issue-1",
explicitRunScopedSkillKeys: ["paperclipai/bundled/software-development/github-pr-workflow"],
})).toBe(false);
});
});
describe("stripHostWorkspaceProvisionForLowTrustSandbox", () => {
it("removes only the host-side provision command for sandbox-backed low-trust runs", () => {
const config = {
@@ -61,12 +61,13 @@ async function waitFor(condition: () => boolean | Promise<boolean>, timeoutMs =
throw new Error("Timed out waiting for condition");
}
async function deleteHeartbeatRunsAfterActivityLogDrains(db: Db) {
async function deleteHeartbeatRunsAndWakeupsAfterActivityLogDrains(db: Db) {
let lastError: unknown = null;
for (let attempt = 0; attempt < 10; attempt += 1) {
await db.delete(activityLog);
try {
await db.delete(heartbeatRuns);
await db.delete(agentWakeupRequests);
return;
} catch (error) {
lastError = error;
@@ -527,8 +528,7 @@ describeEmbeddedPostgres("low-trust red-team HTTP route regression suite", () =>
await db.delete(issueRelations);
await db.delete(activityLog);
await db.delete(heartbeatRunEvents);
await deleteHeartbeatRunsAfterActivityLogDrains(db);
await db.delete(agentWakeupRequests);
await deleteHeartbeatRunsAndWakeupsAfterActivityLogDrains(db);
await db.delete(issues);
await db.delete(agentRuntimeState);
await db.delete(agents);
+166 -5
View File
@@ -260,6 +260,9 @@ const WORKSPACE_VALIDATION_FAILURE_CODE = "workspace_validation_failed";
const WORKSPACE_VALIDATION_RECOVERY_CAUSE = "workspace_validation_failed";
const CONFIGURATION_INCOMPLETE_FAILURE_CODE = "configuration_incomplete";
const CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE = "configuration_incomplete";
const GITHUB_PR_WORKFLOW_SKILL_KEY = "paperclipai/bundled/software-development/github-pr-workflow";
const GITHUB_PR_WORKFLOW_SKILL_SLUG = "github-pr-workflow";
const PUSH_CAPABILITY_ENV_KEYS = ["GH_TOKEN", "GITHUB_TOKEN"] as const;
// Keep this in sync with local adapters that require a git workspace before launch.
const GIT_SENSITIVE_LOCAL_ADAPTER_TYPES = new Set([
"acpx_local",
@@ -411,6 +414,34 @@ function formatMissingBindingForOperator(missing: MissingRuntimeBinding): string
return `secret ${secretLabel} not bound at ${missing.consumerType} ${missing.configPath}`;
}
function isConfiguredEnvBindingValue(binding: unknown) {
const parsed = envBindingSchema.safeParse(binding);
if (!parsed.success) return false;
const value = parsed.data;
if (typeof value === "string") return value.trim().length > 0;
if (value.type === "plain") return value.value.trim().length > 0;
return true;
}
function hasGithubPrWorkflowSkill(desiredSkills: string[]) {
return desiredSkills.some((skill) => {
const normalized = skill.trim();
return normalized === GITHUB_PR_WORKFLOW_SKILL_KEY
|| normalized === GITHUB_PR_WORKFLOW_SKILL_SLUG
|| normalized.endsWith(`/${GITHUB_PR_WORKFLOW_SKILL_SLUG}`);
});
}
export function requiresPushCapabilityPreflight(input: {
adapterType: string;
issueId: string | null | undefined;
explicitRunScopedSkillKeys: string[];
}) {
return Boolean(input.issueId)
&& GIT_SENSITIVE_LOCAL_ADAPTER_TYPES.has(input.adapterType)
&& hasGithubPrWorkflowSkill(input.explicitRunScopedSkillKeys);
}
const LOW_TRUST_SENSITIVE_ENV_KEY_RE =
/(api[-_]?key|access[-_]?token|auth(?:_?token)?|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring)/i;
@@ -466,11 +497,18 @@ export async function resolveExecutionRunAdapterConfig(input: {
routineEnv?: unknown;
secretsSvc: RuntimeConfigSecretResolver;
trustPreset?: TrustPresetResolution;
requiredScopedEnvBinding?: {
keys: string[];
consumerScopes: Array<"agent" | "project">;
reason: string;
remediation: string;
};
}) {
const executionRunConfig = stripPaperclipRuntimeEnvFromAdapterConfig(input.executionRunConfig);
const environmentEnv = stripPaperclipRuntimeEnvBindings(input.environmentEnv);
const projectEnv = stripPaperclipRuntimeEnvBindings(input.projectEnv);
const routineEnv = stripPaperclipRuntimeEnvBindings(input.routineEnv);
const agentEnv = parseObject(executionRunConfig.env);
const lowTrustAllowedBindingIds = input.trustPreset?.kind === "low_trust_review"
? input.trustPreset.boundary.allowedSecretBindingIds ?? []
: undefined;
@@ -480,6 +518,31 @@ export async function resolveExecutionRunAdapterConfig(input: {
assertLowTrustEnvConfigAllowed(projectEnv, "project.env");
assertLowTrustEnvConfigAllowed(routineEnv, "routine.env");
}
const requiredScopedEnvBinding = input.requiredScopedEnvBinding ?? null;
const requiredScopedBindingsConfigured = requiredScopedEnvBinding
? requiredScopedEnvBinding.keys.some((key) => (
requiredScopedEnvBinding.consumerScopes.includes("agent")
&& isConfiguredEnvBindingValue(agentEnv[key])
) || (
requiredScopedEnvBinding.consumerScopes.includes("project")
&& isConfiguredEnvBindingValue(projectEnv?.[key])
))
: false;
if (requiredScopedEnvBinding && !requiredScopedBindingsConfigured) {
throw new ConfigurationIncompleteFailure(`configuration incomplete: ${requiredScopedEnvBinding.remediation}`, {
configurationIncomplete: {
reason: requiredScopedEnvBinding.reason,
companyId: input.companyId,
agentId: input.agentId ?? null,
issueId: input.issueId ?? null,
projectId: input.projectId ?? null,
routineId: input.routineId ?? null,
requiredEnvKeys: requiredScopedEnvBinding.keys,
requiredScopes: requiredScopedEnvBinding.consumerScopes,
missingBindings: [],
},
});
}
// Pre-dispatch binding-validation gate: detect declared secret refs that have
// no binding before resolving any secret value. Missing bindings short-circuit
// to a configuration-incomplete blocker routed to a human owner instead of a
@@ -522,6 +585,33 @@ export async function resolveExecutionRunAdapterConfig(input: {
)),
);
}
if (requiredScopedEnvBinding) {
const requiredEnvKeys = new Set(requiredScopedEnvBinding.keys);
const requiredScopes = new Set(requiredScopedEnvBinding.consumerScopes);
const requiredMissingBindings = missingBindings.filter((binding) =>
requiredScopes.has(binding.consumerType as "agent" | "project")
&& requiredEnvKeys.has(binding.envKey),
);
if (requiredMissingBindings.length > 0) {
const detail = requiredMissingBindings.map(formatMissingBindingForOperator).join("; ");
throw new ConfigurationIncompleteFailure(
`configuration incomplete: ${requiredScopedEnvBinding.remediation}; ${detail}`,
{
configurationIncomplete: {
reason: requiredScopedEnvBinding.reason,
companyId: input.companyId,
agentId: input.agentId ?? null,
issueId: input.issueId ?? null,
projectId: input.projectId ?? null,
routineId: input.routineId ?? null,
requiredEnvKeys: requiredScopedEnvBinding.keys,
requiredScopes: requiredScopedEnvBinding.consumerScopes,
missingBindings: requiredMissingBindings,
},
},
);
}
}
if (missingBindings.length > 0) {
const detail = missingBindings.map(formatMissingBindingForOperator).join("; ");
throw new ConfigurationIncompleteFailure(`configuration incomplete: ${detail}`, {
@@ -1080,6 +1170,53 @@ function sameResolvedPath(left: string | null | undefined, right: string | null
return path.resolve(leftPath) === path.resolve(rightPath);
}
async function hasGitPushRemote(cwd: string | null | undefined) {
const normalized = readNonEmptyString(cwd);
if (!normalized) return false;
const remoteNames = await execFile("git", ["remote"], { cwd: normalized })
.then((result) =>
result.stdout
.split(/\r?\n/)
.map((value) => value.trim())
.filter((value) => value.length > 0),
)
.catch(() => []);
for (const remoteName of remoteNames) {
const pushUrl = await execFile("git", ["remote", "get-url", "--push", remoteName], { cwd: normalized })
.then((result) => readNonEmptyString(result.stdout))
.catch(() => null);
if (pushUrl) return true;
}
return false;
}
export async function assertPushCapabilityCheckoutValid(input: {
enabled: boolean;
issue: {
id: string;
identifier: string | null;
} | null;
cwd: string | null | undefined;
}) {
if (!input.enabled || !input.issue) return;
const cwd = readNonEmptyString(input.cwd);
if (!cwd) return;
if (await hasGitPushRemote(cwd)) return;
throw new WorkspaceValidationFailure(
`Issue ${input.issue.identifier ?? input.issue.id} requested the GitHub PR workflow, but checkout "${cwd}" has no configured push remote. Bind the run to a writable repo checkout before dispatching the agent.`,
{
workspaceValidation: {
reason: "missing_git_push_remote",
issueId: input.issue.id,
issueIdentifier: input.issue.identifier,
executionWorkspaceCwd: cwd,
requiredEnvKeys: [...PUSH_CAPABILITY_ENV_KEYS],
},
},
);
}
export async function assertGitSensitiveAdapterWorkspaceValid(input: {
adapterType: string;
agentId: string;
@@ -8631,6 +8768,16 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
: selectedEnvironmentId
? await environmentsSvc.getById(selectedEnvironmentId)
: null;
const runScopedMentionedSkillKeys = await resolveRunScopedMentionedSkillKeys({
db,
companyId: agent.companyId,
issueId,
});
const pushCapabilityPreflightRequired = requiresPushCapabilityPreflight({
adapterType: agent.adapterType,
issueId,
explicitRunScopedSkillKeys: runScopedMentionedSkillKeys,
});
const { resolvedConfig, secretKeys, secretManifest } = await resolveExecutionRunAdapterConfig({
companyId: agent.companyId,
agentId: agent.id,
@@ -8645,6 +8792,15 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
routineEnv: routineEnvContext.env,
secretsSvc,
trustPreset,
requiredScopedEnvBinding: pushCapabilityPreflightRequired
? {
keys: [...PUSH_CAPABILITY_ENV_KEYS],
consumerScopes: ["agent", "project"],
reason: "push_write_credential_missing",
remediation:
"GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN bound at project or agent scope.",
}
: undefined,
});
if (secretManifest.length > 0) {
context.paperclipSecrets = {
@@ -8653,11 +8809,6 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
} else {
delete context.paperclipSecrets;
}
const runScopedMentionedSkillKeys = await resolveRunScopedMentionedSkillKeys({
db,
companyId: agent.companyId,
issueId,
});
const effectiveResolvedConfig = applyRunScopedMentionedSkillKeys(
resolvedConfig,
runScopedMentionedSkillKeys,
@@ -9271,6 +9422,16 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
environmentDriver: selectedEnvironment.driver,
leaseMetadata: activeEnvironmentLease.lease.metadata,
});
await assertPushCapabilityCheckoutValid({
enabled: pushCapabilityPreflightRequired && executionTarget?.kind === "local",
issue: issueRef
? {
id: issueRef.id,
identifier: issueRef.identifier,
}
: null,
cwd: executionWorkspace.cwd,
});
const adapterEnv = Object.fromEntries(
Object.entries(parseObject(resolvedConfig.env)).filter(
(entry): entry is [string, string] => typeof entry[0] === "string" && typeof entry[1] === "string",