Fix sandbox git publishing and large workspace uploads (#8422)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox and SSH runtimes need to preserve agent work across isolated execution environments > - Git-backed workspaces were being copied mostly as filesystem archives, which breaks when `.git` points outside the mounted workspace and makes sandbox agents unable to publish their own branches > - Large ignored dependency trees could also be swept into the sandbox overlay, causing multi-GB transfers and max-string failures in some sandbox clients > - This pull request makes sandbox runtime setup use a git-backed HEAD sync plus a small dirty/untracked overlay, and bounds sandbox file transfers so large archives do not need one huge string > - The benefit is that sandbox agents can commit and push from a usable git checkout without uploading dependency trees such as `node_modules` ## Linked Issues or Issue Description Refs #8395 No public duplicate issue or PR was found after searches for `sandbox git workspace`, `git push sandbox`, and `node_modules sandbox upload`. Bug report: - What happened: sandbox-backed agent workspaces could receive a `.git` file that pointed at host-only git state, leaving the sandbox unable to run normal git workflows. The sandbox overlay upload could also include ignored dependency directories, creating very large transfers. - Expected behavior: sandbox and remote runtimes should prepare a usable git-backed workspace, copy only the necessary workspace overlay, and restore git history plus file changes without depending on a host-only `.git` path. - Steps to reproduce: 1. Run an agent in a sandbox-backed workspace whose local git checkout is a worktree. 2. Ask the agent to complete a GitHub workflow that requires commit/push access. 3. Observe that git operations can fail inside the sandbox, and ignored dependency trees can be uploaded as part of the workspace overlay. - Paperclip version or commit: reproduced against `master` before this PR, base `7aa212296eb1`. - Deployment mode: local dev / sandbox-backed runtime. - Installation method: built from source. - Agent adapters involved: local adapters using shared adapter-utils runtime preparation. - Database mode: not database-related. - Access context: agent runtime. - Local verification environment: Node.js v25.6.1, pnpm 9.15.4, macOS arm64. - Privacy checklist: all pasted output was reviewed for secrets, private hostnames, local usernames, and internal instance links. ## What Changed - Added a GitHub workflow push preflight so agent runs can detect missing push credentials when a workflow explicitly needs GitHub publishing. - Added shared git workspace sync helpers for shallow HEAD import/export and dirty/untracked overlay tracking. - Updated sandbox managed runtime setup to use git history plus a selected overlay instead of uploading the full local workspace for git-backed workspaces. - Bounded sandbox archive upload/download paths so large payloads stream or chunk instead of materializing one oversized string. - Excluded `.git` and ignored dependency trees from sandbox upload, download, and restore baselines while preserving local ignored directories during sync-back. - Added focused tests for git workspace sync, sandbox overlay selection, transfer chunking, and heartbeat push-preflight behavior. ## Verification - `pnpm exec vitest run packages/adapter-utils/src/git-workspace-sync.test.ts packages/adapter-utils/src/sandbox-managed-runtime.test.ts packages/adapter-utils/src/command-managed-runtime.test.ts server/src/__tests__/heartbeat-project-env.test.ts server/src/__tests__/heartbeat-workspace-session.test.ts` - `pnpm --filter @paperclipai/adapter-utils typecheck` - `pnpm --filter @paperclipai/server typecheck` - `pnpm build` - Public-hygiene scan of the PR diff and commit messages for internal issue ids, local paths, private hostnames, and obvious token patterns. ## Risks - Medium risk: this changes sandbox runtime synchronization semantics for git-backed workspaces, especially around dirty tracked files, untracked files, deleted paths, and ignored files. - The main mitigation is focused test coverage for upload contents, restore exclusions, and git round-trip behavior. - The SSH runtime keeps the current bundle-based implementation from `master`; this PR only aligns shared excludes and sandbox behavior with that model. ## Model Used OpenAI Codex, GPT-5-based coding agent, tool-enabled shell/git/GitHub workflow, with code execution and repository inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
@@ -22,7 +22,10 @@ interface SpawnRunnerHandle {
|
||||
// A runner that actually executes the shell scripts (piping stdin through a real
|
||||
// pipe so multi-MB payloads work) and replays stdout through onLog in several
|
||||
// chunks so the streaming readFile byte-counter is exercised.
|
||||
function makeSpawnRunner(options: { supportsSingleStreamStdinProgress?: boolean } = {}): SpawnRunnerHandle {
|
||||
function makeSpawnRunner(options: {
|
||||
supportsSingleStreamStdinProgress?: boolean;
|
||||
maxStdoutBytes?: number;
|
||||
} = {}): SpawnRunnerHandle {
|
||||
const calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }> = [];
|
||||
const runner: CommandManagedRuntimeRunner = {
|
||||
supportsSingleStreamStdinProgress: options.supportsSingleStreamStdinProgress,
|
||||
@@ -48,6 +51,21 @@ function makeSpawnRunner(options: { supportsSingleStreamStdinProgress?: boolean
|
||||
resolve({ exitCode: 127, signal: null, timedOut: false, stdout, stderr, pid: null, startedAt });
|
||||
});
|
||||
child.on("close", async (code) => {
|
||||
if (
|
||||
options.maxStdoutBytes != null &&
|
||||
Buffer.byteLength(stdout, "utf8") > options.maxStdoutBytes
|
||||
) {
|
||||
resolve({
|
||||
exitCode: 1,
|
||||
signal: null,
|
||||
timedOut: false,
|
||||
stdout,
|
||||
stderr: `stdout exceeded ${options.maxStdoutBytes} bytes`,
|
||||
pid: child.pid ?? null,
|
||||
startedAt,
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (input.onLog && stdout.length > 0) {
|
||||
const chunkSize = Math.max(1, Math.ceil(stdout.length / 4));
|
||||
for (let offset = 0; offset < stdout.length; offset += chunkSize) {
|
||||
@@ -75,6 +93,26 @@ function toArrayBuffer(buffer: Buffer): ArrayBuffer {
|
||||
return Uint8Array.from(buffer).buffer;
|
||||
}
|
||||
|
||||
async function withBase64StringByteLimit<T>(limitBytes: number, fn: () => Promise<T>): Promise<T> {
|
||||
const originalToString = Buffer.prototype.toString;
|
||||
Buffer.prototype.toString = function patchedToString(
|
||||
this: Buffer,
|
||||
encoding?: BufferEncoding,
|
||||
start?: number,
|
||||
end?: number,
|
||||
) {
|
||||
if (encoding === "base64" && this.byteLength > limitBytes) {
|
||||
throw new Error(`test guard: attempted to base64-encode ${this.byteLength} bytes at once`);
|
||||
}
|
||||
return originalToString.call(this, encoding, start, end);
|
||||
} as typeof Buffer.prototype.toString;
|
||||
try {
|
||||
return await fn();
|
||||
} finally {
|
||||
Buffer.prototype.toString = originalToString;
|
||||
}
|
||||
}
|
||||
|
||||
describe("command managed runtime", () => {
|
||||
const cleanupDirs: string[] = [];
|
||||
|
||||
@@ -243,10 +281,12 @@ describe("command managed runtime", () => {
|
||||
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
||||
|
||||
const progress: Array<{ done: number; total: number | null }> = [];
|
||||
await client.writeFile(remotePath, toArrayBuffer(payload), {
|
||||
onProgress: (done, total) => {
|
||||
progress.push({ done, total });
|
||||
},
|
||||
await withBase64StringByteLimit(4 * 1024 * 1024, async () => {
|
||||
await client.writeFile(remotePath, toArrayBuffer(payload), {
|
||||
onProgress: (done, total) => {
|
||||
progress.push({ done, total });
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
// Exactly one upload process: O(1) round-trips regardless of payload size.
|
||||
@@ -288,6 +328,9 @@ describe("command managed runtime", () => {
|
||||
// Provider-backed sandbox runners cannot surface mid-flight progress for a
|
||||
// single stdin RPC, so we intentionally use several large append commands.
|
||||
expect(calls.length).toBeGreaterThan(2);
|
||||
const stdinCalls = calls.filter((call) => call.stdin != null);
|
||||
expect(stdinCalls.length).toBeGreaterThan(2);
|
||||
expect(stdinCalls.every((call) => Buffer.byteLength(call.stdin ?? "", "utf8") <= 4.1 * 1024 * 1024)).toBe(true);
|
||||
expect(progress.length).toBeGreaterThan(2);
|
||||
for (let i = 1; i < progress.length; i++) {
|
||||
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
|
||||
@@ -295,16 +338,41 @@ describe("command managed runtime", () => {
|
||||
expect(progress.at(-1)).toEqual({ done: payload.length, total: payload.length });
|
||||
});
|
||||
|
||||
it("streams a download through onLog and reports monotonic byte progress to the total", async () => {
|
||||
it("falls back to bounded chunks when the runner does not explicitly opt in", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-write-fallback-no-progress-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const remotePath = path.join(rootDir, "nested", "payload.bin");
|
||||
|
||||
const payload = Buffer.alloc(12 * 1024 * 1024);
|
||||
for (let i = 0; i < payload.length; i++) payload[i] = i % 256;
|
||||
|
||||
const { runner, calls } = makeSpawnRunner();
|
||||
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
||||
|
||||
await withBase64StringByteLimit(4 * 1024 * 1024, async () => {
|
||||
await client.writeFile(remotePath, toArrayBuffer(payload));
|
||||
});
|
||||
|
||||
const written = await readFile(remotePath);
|
||||
expect(written.equals(payload)).toBe(true);
|
||||
|
||||
// A runner that doesn't mark single-stream stdin support must avoid passing
|
||||
// the whole base64 archive as one string, so we expect multiple append calls.
|
||||
const stdinCalls = calls.filter((call) => call.stdin != null);
|
||||
expect(stdinCalls.length).toBeGreaterThan(1);
|
||||
expect(stdinCalls.every((call) => Buffer.byteLength(call.stdin ?? "", "utf8") <= 4.1 * 1024 * 1024)).toBe(true);
|
||||
});
|
||||
|
||||
it("downloads in bounded stdout chunks and reports monotonic byte progress to the total", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-read-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const remotePath = path.join(rootDir, "download.bin");
|
||||
|
||||
const payload = Buffer.alloc(512 * 1024);
|
||||
const payload = Buffer.alloc(7 * 1024 * 1024);
|
||||
for (let i = 0; i < payload.length; i++) payload[i] = (i * 7) % 256;
|
||||
await writeFile(remotePath, payload);
|
||||
|
||||
const { runner } = makeSpawnRunner();
|
||||
const { runner, calls } = makeSpawnRunner({ maxStdoutBytes: 5 * 1024 * 1024 });
|
||||
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
||||
|
||||
const progress: Array<{ done: number; total: number | null }> = [];
|
||||
@@ -316,7 +384,10 @@ describe("command managed runtime", () => {
|
||||
|
||||
expect(Buffer.from(bytes as ArrayBuffer).equals(payload)).toBe(true);
|
||||
|
||||
// The runner replays stdout in several chunks, so progress arrives in steps.
|
||||
// The old single `base64 < file` path would exceed the runner's stdout cap.
|
||||
// The bounded path reads with several small `dd | base64` commands instead.
|
||||
expect(calls.some((call) => call.args?.join(" ").includes("base64 <"))).toBe(false);
|
||||
expect(calls.filter((call) => call.args?.join(" ").includes("dd if=")).length).toBeGreaterThan(1);
|
||||
expect(progress.length).toBeGreaterThan(1);
|
||||
for (let i = 1; i < progress.length; i++) {
|
||||
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
|
||||
|
||||
@@ -56,6 +56,12 @@ const REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES = 96 * 1024 * 1024;
|
||||
// Fallback chunk size (base64 bytes). Kept a multiple of 4 so each chunk is a
|
||||
// self-contained base64 unit that decodes cleanly on its own.
|
||||
const REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE = 4 * 1024 * 1024;
|
||||
const REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE = (REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE / 4) * 3;
|
||||
const REMOTE_READ_CHUNK_BYTES = REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE;
|
||||
|
||||
function base64EncodedLength(byteLength: number): number {
|
||||
return Math.ceil(byteLength / 3) * 4;
|
||||
}
|
||||
|
||||
function toBuffer(bytes: Buffer | Uint8Array | ArrayBuffer): Buffer {
|
||||
if (Buffer.isBuffer(bytes)) return bytes;
|
||||
@@ -104,11 +110,10 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
writeFile: async (remotePath, bytes, options) => {
|
||||
const buffer = toBuffer(bytes);
|
||||
const total = buffer.byteLength;
|
||||
const body = buffer.toString("base64");
|
||||
const encodedLength = base64EncodedLength(total);
|
||||
const remoteDir = path.posix.dirname(remotePath);
|
||||
const remoteTempPath = `${remotePath}.paperclip-upload`;
|
||||
const canUseSingleStreamProgressPath =
|
||||
!options?.onProgress || input.runner.supportsSingleStreamStdinProgress === true;
|
||||
const canUseSingleStreamProgressPath = input.runner.supportsSingleStreamStdinProgress === true;
|
||||
|
||||
// Primary path: a single round-trip. Stream the entire base64 body to one
|
||||
// `base64 -d` process via stdin, decode straight into a temp file, then
|
||||
@@ -116,9 +121,10 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
// one `printf >> tmpfile` shell round-trip per 32 KB — thousands of serial
|
||||
// processes for a large workspace — with exactly one process.
|
||||
if (
|
||||
body.length <= REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES &&
|
||||
encodedLength <= REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES &&
|
||||
canUseSingleStreamProgressPath
|
||||
) {
|
||||
const body = buffer.toString("base64");
|
||||
await options?.onProgress?.(0, total);
|
||||
await runShell(
|
||||
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
||||
@@ -139,62 +145,50 @@ export function createCommandManagedRuntimeClient(input: {
|
||||
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
||||
`rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`,
|
||||
);
|
||||
for (let offset = 0; offset < body.length; offset += REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE) {
|
||||
const chunk = body.slice(offset, offset + REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE);
|
||||
for (let offset = 0; offset < total; offset += REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE) {
|
||||
const end = Math.min(total, offset + REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE);
|
||||
const chunk = buffer.subarray(offset, end).toString("base64");
|
||||
await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk });
|
||||
const decodedSoFar = Math.min(total, Math.floor(((offset + chunk.length) * 3) / 4));
|
||||
await options?.onProgress?.(decodedSoFar, total);
|
||||
await options?.onProgress?.(end, total);
|
||||
}
|
||||
await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`);
|
||||
await options?.onProgress?.(total, total);
|
||||
},
|
||||
readFile: async (remotePath, options) => {
|
||||
// Decoded file size up front so download progress can be a percentage.
|
||||
// Only paid when a progress hook is attached.
|
||||
let totalBytes: number | null = null;
|
||||
if (options?.onProgress) {
|
||||
const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`);
|
||||
const parsed = Number.parseInt(sizeResult.stdout.trim(), 10);
|
||||
totalBytes = Number.isFinite(parsed) && parsed >= 0 ? parsed : null;
|
||||
// Chunked reads intentionally query the remote size first, even without
|
||||
// a progress sink, so each sandbox RPC stays bounded and truncation is
|
||||
// detected without materializing the whole file as one stdout string.
|
||||
const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`);
|
||||
const totalBytes = Number.parseInt(sizeResult.stdout.trim(), 10);
|
||||
if (!Number.isFinite(totalBytes) || totalBytes < 0) {
|
||||
throw new Error(`Could not determine remote file size for ${remotePath}`);
|
||||
}
|
||||
|
||||
// Stream the remote `base64` stdout through a byte counter, decoding each
|
||||
// 4-char-aligned slice incrementally rather than buffering the whole
|
||||
// response as one string. Falls back to the buffered result when the
|
||||
// runner does not surface incremental stdout.
|
||||
// Read in bounded remote chunks so the runner never has to materialize a
|
||||
// single base64 stdout string for the whole archive. The client API still
|
||||
// returns the decoded file as a Buffer, but every command result stays
|
||||
// small enough for provider-backed sandbox RPCs.
|
||||
const decodedChunks: Buffer[] = [];
|
||||
let b64Remainder = "";
|
||||
let decodedSoFar = 0;
|
||||
let streamed = false;
|
||||
const result = await runShell(`base64 < ${shellQuote(remotePath)}`, {
|
||||
onLog: async (stream, chunk) => {
|
||||
if (stream !== "stdout") return;
|
||||
streamed = true;
|
||||
const data = b64Remainder + chunk.replace(/\s+/g, "");
|
||||
const alignedLen = data.length - (data.length % 4);
|
||||
if (alignedLen > 0) {
|
||||
const decoded = Buffer.from(data.slice(0, alignedLen), "base64");
|
||||
decodedChunks.push(decoded);
|
||||
decodedSoFar += decoded.byteLength;
|
||||
}
|
||||
b64Remainder = data.slice(alignedLen);
|
||||
if (options?.onProgress) {
|
||||
const done = totalBytes != null ? Math.min(totalBytes, decodedSoFar) : decodedSoFar;
|
||||
await options.onProgress(done, totalBytes);
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
let out: Buffer;
|
||||
if (streamed) {
|
||||
if (b64Remainder.length > 0) {
|
||||
decodedChunks.push(Buffer.from(b64Remainder, "base64"));
|
||||
}
|
||||
out = Buffer.concat(decodedChunks);
|
||||
} else {
|
||||
out = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
|
||||
if (totalBytes === 0) {
|
||||
await options?.onProgress?.(0, 0);
|
||||
return Buffer.alloc(0);
|
||||
}
|
||||
await options?.onProgress?.(out.byteLength, totalBytes ?? out.byteLength);
|
||||
for (let chunkIndex = 0; decodedSoFar < totalBytes; chunkIndex++) {
|
||||
const result = await runShell(
|
||||
`dd if=${shellQuote(remotePath)} bs=${REMOTE_READ_CHUNK_BYTES} skip=${chunkIndex} count=1 2>/dev/null | base64`,
|
||||
);
|
||||
const chunk = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
|
||||
if (chunk.byteLength === 0) break;
|
||||
decodedChunks.push(chunk);
|
||||
decodedSoFar += chunk.byteLength;
|
||||
await options?.onProgress?.(Math.min(decodedSoFar, totalBytes), totalBytes);
|
||||
}
|
||||
const out = Buffer.concat(decodedChunks);
|
||||
if (out.byteLength !== totalBytes) {
|
||||
throw new Error(`Remote file read was truncated for ${remotePath}: ${out.byteLength}/${totalBytes} bytes`);
|
||||
}
|
||||
await options?.onProgress?.(out.byteLength, totalBytes);
|
||||
return out;
|
||||
},
|
||||
listFiles: async (remotePath) => {
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
export function isRelativePathOrDescendant(relative: string, candidate: string): boolean {
|
||||
return relative === candidate || relative.startsWith(`${candidate}/`);
|
||||
}
|
||||
|
||||
function pathContainsSegmentOrDescendant(relative: string, segment: string): boolean {
|
||||
return relative === segment ||
|
||||
relative.startsWith(`${segment}/`) ||
|
||||
relative.endsWith(`/${segment}`) ||
|
||||
relative.includes(`/${segment}/`);
|
||||
}
|
||||
|
||||
export function excludePatternMatches(relative: string, pattern: string): boolean {
|
||||
if (pattern.startsWith("*/") && pattern.endsWith("/*")) {
|
||||
return pathContainsSegmentOrDescendant(relative, pattern.slice(2, -2));
|
||||
}
|
||||
if (pattern.startsWith("*/")) {
|
||||
return pathContainsSegmentOrDescendant(relative, pattern.slice(2));
|
||||
}
|
||||
if (pattern.endsWith("/*")) {
|
||||
const base = pattern.slice(0, -2);
|
||||
return relative.startsWith(`${base}/`);
|
||||
}
|
||||
return isRelativePathOrDescendant(relative, pattern);
|
||||
}
|
||||
|
||||
export function shouldExcludePath(relative: string, exclude: readonly string[]): boolean {
|
||||
return exclude.some((entry) => excludePatternMatches(relative, entry));
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import { lstat, mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
buildRemoteGitDeltaBundleScript,
|
||||
createImportedGitRef,
|
||||
createRemoteGitExportRef,
|
||||
deleteLocalGitRef,
|
||||
fetchGitBundleIntoLocalRef,
|
||||
readGitWorkspaceSnapshot,
|
||||
runLocalGit,
|
||||
withShallowGitWorkspaceClone,
|
||||
} from "./git-workspace-sync.js";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
|
||||
async function git(cwd: string, args: string[]): Promise<string> {
|
||||
return (await runLocalGit(cwd, args)).stdout.trim();
|
||||
}
|
||||
|
||||
describe("git workspace sync", () => {
|
||||
const cleanupDirs: string[] = [];
|
||||
|
||||
afterEach(async () => {
|
||||
while (cleanupDirs.length > 0) {
|
||||
const dir = cleanupDirs.pop();
|
||||
if (!dir) continue;
|
||||
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
|
||||
}
|
||||
});
|
||||
|
||||
async function createRepo(rootDir: string): Promise<string> {
|
||||
const repo = path.join(rootDir, "repo");
|
||||
await mkdir(repo, { recursive: true });
|
||||
await git(repo, ["init"]);
|
||||
await git(repo, ["checkout", "-b", "main"]);
|
||||
await git(repo, ["config", "user.name", "Paperclip Test"]);
|
||||
await git(repo, ["config", "user.email", "test@paperclip.dev"]);
|
||||
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
|
||||
await git(repo, ["add", "tracked.txt"]);
|
||||
await git(repo, ["commit", "-m", "base"]);
|
||||
return repo;
|
||||
}
|
||||
|
||||
it("creates a shallow standalone clone from the local HEAD snapshot", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-sync-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const repo = await createRepo(rootDir);
|
||||
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
|
||||
await rm(path.join(repo, "tracked.txt"));
|
||||
|
||||
const snapshot = await readGitWorkspaceSnapshot(repo);
|
||||
expect(snapshot).toMatchObject({
|
||||
headCommit: baseHead,
|
||||
branchName: "main",
|
||||
deletedPaths: ["tracked.txt"],
|
||||
});
|
||||
|
||||
await withShallowGitWorkspaceClone({
|
||||
localDir: repo,
|
||||
snapshot: snapshot!,
|
||||
}, async (cloneDir) => {
|
||||
expect((await lstat(path.join(cloneDir, ".git"))).isDirectory()).toBe(true);
|
||||
await expect(readFile(path.join(cloneDir, ".git", "shallow"), "utf8")).resolves.toContain(baseHead);
|
||||
expect(await git(cloneDir, ["rev-list", "--count", "HEAD"])).toBe("1");
|
||||
expect(await git(cloneDir, ["branch", "--show-current"])).toBe("main");
|
||||
await expect(readFile(path.join(cloneDir, "tracked.txt"), "utf8")).resolves.toBe("base\n");
|
||||
});
|
||||
});
|
||||
|
||||
it("builds thin git delta bundles relative to the imported base", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-delta-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const repo = await createRepo(rootDir);
|
||||
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
|
||||
const snapshot = await readGitWorkspaceSnapshot(repo);
|
||||
expect(snapshot).not.toBeNull();
|
||||
|
||||
await withShallowGitWorkspaceClone({
|
||||
localDir: repo,
|
||||
snapshot: snapshot!,
|
||||
}, async (remoteDir) => {
|
||||
const emptyBundle = path.join(rootDir, "empty.bundle");
|
||||
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
|
||||
remoteDir,
|
||||
baseSha: baseHead,
|
||||
exportRef: createRemoteGitExportRef("test"),
|
||||
bundlePath: emptyBundle,
|
||||
})]);
|
||||
expect((await stat(emptyBundle)).size).toBe(0);
|
||||
|
||||
await git(remoteDir, ["config", "user.name", "Paperclip Remote"]);
|
||||
await git(remoteDir, ["config", "user.email", "remote@paperclip.dev"]);
|
||||
await writeFile(path.join(remoteDir, "tracked.txt"), "remote\n", "utf8");
|
||||
await git(remoteDir, ["commit", "-am", "remote update"]);
|
||||
const remoteHead = await git(remoteDir, ["rev-parse", "HEAD"]);
|
||||
|
||||
const deltaBundle = path.join(rootDir, "delta.bundle");
|
||||
const importedRef = createImportedGitRef("test");
|
||||
const exportRef = createRemoteGitExportRef("test");
|
||||
try {
|
||||
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
|
||||
remoteDir,
|
||||
baseSha: baseHead,
|
||||
exportRef,
|
||||
bundlePath: deltaBundle,
|
||||
})]);
|
||||
expect((await stat(deltaBundle)).size).toBeGreaterThan(0);
|
||||
|
||||
const importedHead = await fetchGitBundleIntoLocalRef({
|
||||
localDir: repo,
|
||||
bundlePath: deltaBundle,
|
||||
exportRef,
|
||||
importedRef,
|
||||
baseSha: baseHead,
|
||||
});
|
||||
expect(importedHead).toBe(remoteHead);
|
||||
expect(await git(repo, ["rev-list", "--count", importedRef, "--not", baseHead])).toBe("1");
|
||||
} finally {
|
||||
await deleteLocalGitRef({ localDir: repo, ref: importedRef });
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,321 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { execFile } from "node:child_process";
|
||||
import { promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
export interface GitCommandResult {
|
||||
stdout: string;
|
||||
stderr: string;
|
||||
}
|
||||
|
||||
export interface GitWorkspaceSnapshot {
|
||||
headCommit: string;
|
||||
branchName: string | null;
|
||||
overlayPaths: string[];
|
||||
deletedPaths: string[];
|
||||
ignoredPaths: string[];
|
||||
}
|
||||
|
||||
export const GIT_ARCHIVE_EXCLUDES = [".git", ".git/*"] as const;
|
||||
|
||||
function shellQuote(value: string) {
|
||||
return `'${value.replace(/'/g, `'\"'\"'`)}'`;
|
||||
}
|
||||
|
||||
export async function runLocalGit(
|
||||
localDir: string,
|
||||
args: string[],
|
||||
options: {
|
||||
timeout?: number;
|
||||
maxBuffer?: number;
|
||||
} = {},
|
||||
): Promise<GitCommandResult> {
|
||||
return await new Promise<GitCommandResult>((resolve, reject) => {
|
||||
execFile(
|
||||
"git",
|
||||
["-C", localDir, ...args],
|
||||
{
|
||||
timeout: options.timeout ?? 15_000,
|
||||
maxBuffer: options.maxBuffer ?? 1024 * 128,
|
||||
},
|
||||
(error, stdout, stderr) => {
|
||||
if (error) {
|
||||
reject(Object.assign(error, { stdout: stdout ?? "", stderr: stderr ?? "" }));
|
||||
return;
|
||||
}
|
||||
resolve({
|
||||
stdout: stdout ?? "",
|
||||
stderr: stderr ?? "",
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
export async function readGitWorkspaceSnapshot(localDir: string): Promise<GitWorkspaceSnapshot | null> {
|
||||
try {
|
||||
const insideWorkTree = await runLocalGit(localDir, ["rev-parse", "--is-inside-work-tree"], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
});
|
||||
if (insideWorkTree.stdout.trim() !== "true") {
|
||||
return null;
|
||||
}
|
||||
|
||||
const [headCommitResult, branchResult, overlayDiffResult, untrackedResult, deletedResult, ignoredResult] = await Promise.all([
|
||||
runLocalGit(localDir, ["rev-parse", "HEAD"], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
}),
|
||||
runLocalGit(localDir, ["rev-parse", "--abbrev-ref", "HEAD"], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
}),
|
||||
runLocalGit(localDir, ["diff", "--name-only", "-z", "--diff-filter=ACMRTUXB", "HEAD", "--"], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
}),
|
||||
runLocalGit(localDir, ["ls-files", "--others", "--exclude-standard", "-z"], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
}),
|
||||
runLocalGit(localDir, ["diff", "--name-only", "-z", "--diff-filter=D", "HEAD", "--"], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 256 * 1024,
|
||||
}),
|
||||
runLocalGit(localDir, ["status", "--ignored", "--porcelain=v1", "-z", "--untracked-files=normal"], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
}),
|
||||
]);
|
||||
|
||||
const branchName = branchResult.stdout.trim();
|
||||
const splitNul = (value: string) => value.split("\0").map((entry) => entry.trim()).filter(Boolean);
|
||||
return {
|
||||
headCommit: headCommitResult.stdout.trim(),
|
||||
branchName: branchName && branchName !== "HEAD" ? branchName : null,
|
||||
overlayPaths: [...new Set([...splitNul(overlayDiffResult.stdout), ...splitNul(untrackedResult.stdout)])]
|
||||
.sort((left, right) => left.localeCompare(right)),
|
||||
deletedPaths: [...new Set(splitNul(deletedResult.stdout))]
|
||||
.sort((left, right) => left.localeCompare(right)),
|
||||
ignoredPaths: splitNul(ignoredResult.stdout)
|
||||
.filter((entry) => entry.startsWith("!! "))
|
||||
.map((entry) => entry.slice(3).replace(/\/+$/, ""))
|
||||
.filter(Boolean)
|
||||
.sort((left, right) => left.localeCompare(right)),
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function withShallowGitWorkspaceClone<T>(
|
||||
input: {
|
||||
localDir: string;
|
||||
snapshot: GitWorkspaceSnapshot;
|
||||
},
|
||||
fn: (cloneDir: string) => Promise<T>,
|
||||
): Promise<T> {
|
||||
const cloneDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-workspace-"));
|
||||
const tempRef = `refs/paperclip/git-sync/import/${randomUUID()}`;
|
||||
try {
|
||||
await runLocalGit(input.localDir, ["update-ref", tempRef, input.snapshot.headCommit], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
});
|
||||
await runLocalGit(cloneDir, ["init"], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 64 * 1024,
|
||||
});
|
||||
await runLocalGit(cloneDir, ["fetch", "--depth=1", input.localDir, tempRef], {
|
||||
timeout: 60_000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
});
|
||||
await runLocalGit(
|
||||
cloneDir,
|
||||
input.snapshot.branchName
|
||||
? ["checkout", "--force", "-B", input.snapshot.branchName, "FETCH_HEAD"]
|
||||
: ["checkout", "--force", "--detach", "FETCH_HEAD"],
|
||||
{
|
||||
timeout: 60_000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
},
|
||||
);
|
||||
await runLocalGit(cloneDir, ["reset", "--hard", input.snapshot.headCommit], {
|
||||
timeout: 60_000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
});
|
||||
return await fn(cloneDir);
|
||||
} finally {
|
||||
await runLocalGit(input.localDir, ["update-ref", "-d", tempRef], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
}).catch(() => undefined);
|
||||
await fs.rm(cloneDir, { recursive: true, force: true }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
|
||||
export function createImportedGitRef(scope = "remote"): string {
|
||||
return `refs/paperclip/git-sync/imported/${scope}/${randomUUID()}`;
|
||||
}
|
||||
|
||||
export function createRemoteGitExportRef(scope = "remote"): string {
|
||||
return `refs/paperclip/git-sync/export/${scope}/${randomUUID()}`;
|
||||
}
|
||||
|
||||
export async function deleteLocalGitRef(input: {
|
||||
localDir: string;
|
||||
ref: string;
|
||||
}): Promise<void> {
|
||||
await runLocalGit(input.localDir, ["update-ref", "-d", input.ref], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
}).catch(() => undefined);
|
||||
}
|
||||
|
||||
export async function fetchGitBundleIntoLocalRef(input: {
|
||||
localDir: string;
|
||||
bundlePath: string;
|
||||
exportRef: string;
|
||||
importedRef: string;
|
||||
baseSha: string;
|
||||
}): Promise<string> {
|
||||
const bundleSize = (await fs.stat(input.bundlePath).catch(() => null))?.size ?? 0;
|
||||
if (bundleSize === 0) {
|
||||
return input.baseSha;
|
||||
}
|
||||
|
||||
await runLocalGit(input.localDir, ["fetch", "--force", input.bundlePath, `${input.exportRef}:${input.importedRef}`], {
|
||||
timeout: 60_000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
});
|
||||
const importedHead = await runLocalGit(input.localDir, ["rev-parse", input.importedRef], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
});
|
||||
return importedHead.stdout.trim();
|
||||
}
|
||||
|
||||
export function buildRemoteGitDeltaBundleScript(input: {
|
||||
remoteDir: string;
|
||||
baseSha: string;
|
||||
exportRef: string;
|
||||
bundlePath: string;
|
||||
catBundle?: boolean;
|
||||
cleanupBundle?: boolean;
|
||||
}): string {
|
||||
const remoteDir = shellQuote(input.remoteDir);
|
||||
const bundlePath = shellQuote(input.bundlePath);
|
||||
const exportRef = shellQuote(input.exportRef);
|
||||
const baseSha = shellQuote(input.baseSha);
|
||||
const cleanupParts = [
|
||||
`rm -f ${bundlePath}`,
|
||||
`git -C ${remoteDir} update-ref -d ${exportRef} >/dev/null 2>&1 || true`,
|
||||
];
|
||||
return [
|
||||
"set -e",
|
||||
input.cleanupBundle ? `cleanup() { ${cleanupParts.join("; ")}; }` : "",
|
||||
input.cleanupBundle ? "trap cleanup EXIT" : "",
|
||||
`mkdir -p ${shellQuote(path.posix.dirname(input.bundlePath))}`,
|
||||
`rm -f ${bundlePath}`,
|
||||
`git -C ${remoteDir} cat-file -e ${baseSha}^{commit}`,
|
||||
`commit_count=$(git -C ${remoteDir} rev-list --count HEAD --not ${baseSha})`,
|
||||
'if [ "$commit_count" -gt 0 ]; then',
|
||||
` git -C ${remoteDir} update-ref ${exportRef} HEAD`,
|
||||
` git -C ${remoteDir} bundle create ${bundlePath} ${exportRef} --not ${baseSha} >/dev/null`,
|
||||
"else",
|
||||
` : > ${bundlePath}`,
|
||||
"fi",
|
||||
input.catBundle ? `cat ${bundlePath}` : "",
|
||||
].filter(Boolean).join("\n");
|
||||
}
|
||||
|
||||
export async function integrateImportedGitHead(input: {
|
||||
localDir: string;
|
||||
importedHead: string;
|
||||
}): Promise<void> {
|
||||
const isConcurrentRefUpdateError = (error: unknown) => {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
return message.includes("cannot lock ref") && message.includes("expected");
|
||||
};
|
||||
|
||||
for (let attempt = 0; attempt < 5; attempt += 1) {
|
||||
const snapshot = await readGitWorkspaceSnapshot(input.localDir);
|
||||
if (!snapshot) return;
|
||||
|
||||
const currentHead = snapshot.headCommit;
|
||||
if (!currentHead || currentHead === input.importedHead) return;
|
||||
|
||||
const headRef = snapshot.branchName ? `refs/heads/${snapshot.branchName}` : "HEAD";
|
||||
const mergeBase = await runLocalGit(input.localDir, ["merge-base", currentHead, input.importedHead], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
}).catch(() => null);
|
||||
const mergeBaseHead = mergeBase?.stdout.trim() ?? "";
|
||||
|
||||
if (mergeBaseHead === input.importedHead) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (mergeBaseHead === currentHead) {
|
||||
try {
|
||||
await runLocalGit(input.localDir, ["update-ref", headRef, input.importedHead, currentHead], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
});
|
||||
return;
|
||||
} catch (error) {
|
||||
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
let mergedTree;
|
||||
try {
|
||||
mergedTree = await runLocalGit(input.localDir, ["merge-tree", "--write-tree", currentHead, input.importedHead], {
|
||||
timeout: 60_000,
|
||||
maxBuffer: 256 * 1024,
|
||||
});
|
||||
} catch (error) {
|
||||
const reason = error instanceof Error ? error.message : String(error);
|
||||
throw new Error(
|
||||
`Failed to merge concurrent remote git histories for ${currentHead.slice(0, 12)} and ${input.importedHead.slice(0, 12)}: ${reason}`,
|
||||
);
|
||||
}
|
||||
const mergedTreeId = mergedTree.stdout.trim().split("\n")[0]?.trim() ?? "";
|
||||
if (!mergedTreeId) {
|
||||
throw new Error("Failed to compute a merged git tree for workspace restore.");
|
||||
}
|
||||
|
||||
const mergeCommit = await runLocalGit(
|
||||
input.localDir,
|
||||
[
|
||||
"commit-tree",
|
||||
mergedTreeId,
|
||||
"-p",
|
||||
currentHead,
|
||||
"-p",
|
||||
input.importedHead,
|
||||
"-m",
|
||||
`Paperclip remote git sync merge ${input.importedHead.slice(0, 12)}`,
|
||||
],
|
||||
{
|
||||
timeout: 60_000,
|
||||
maxBuffer: 64 * 1024,
|
||||
},
|
||||
);
|
||||
try {
|
||||
await runLocalGit(input.localDir, ["update-ref", headRef, mergeCommit.stdout.trim(), currentHead], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 16 * 1024,
|
||||
});
|
||||
return;
|
||||
} catch (error) {
|
||||
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
throw new Error(`Failed to integrate concurrent remote git history for ${input.importedHead.slice(0, 12)} after multiple retries.`);
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
import path from "node:path";
|
||||
import { GIT_ARCHIVE_EXCLUDES } from "./git-workspace-sync.js";
|
||||
import {
|
||||
type SshRemoteExecutionSpec,
|
||||
prepareWorkspaceForSshExecution,
|
||||
@@ -90,7 +91,7 @@ export async function prepareRemoteManagedRuntime(input: {
|
||||
remoteDir: workspaceRemoteDir,
|
||||
onProgress: input.onProgress,
|
||||
});
|
||||
const restoreExclude = preparedWorkspace.gitBacked ? [".git", ".paperclip-runtime"] : [".paperclip-runtime"];
|
||||
const restoreExclude = preparedWorkspace.gitBacked ? [...GIT_ARCHIVE_EXCLUDES, ".paperclip-runtime"] : [".paperclip-runtime"];
|
||||
const baselineSnapshot = await captureDirectorySnapshot(input.workspaceLocalDir, {
|
||||
exclude: restoreExclude,
|
||||
});
|
||||
|
||||
@@ -13,6 +13,20 @@ import {
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
|
||||
async function git(cwd: string, args: string[]): Promise<string> {
|
||||
const { stdout } = await execFile("git", ["-C", cwd, ...args], {
|
||||
maxBuffer: 32 * 1024 * 1024,
|
||||
});
|
||||
return stdout.trim();
|
||||
}
|
||||
|
||||
async function listTarMembers(rootDir: string, name: string, bytes: Buffer): Promise<string[]> {
|
||||
const tarPath = path.join(rootDir, name);
|
||||
await writeFile(tarPath, bytes);
|
||||
const { stdout } = await execFile("tar", ["-tf", tarPath], { maxBuffer: 32 * 1024 * 1024 });
|
||||
return stdout.split("\n").map((line) => line.trim()).filter(Boolean);
|
||||
}
|
||||
|
||||
describe("sandbox managed runtime", () => {
|
||||
const cleanupDirs: string[] = [];
|
||||
|
||||
@@ -131,6 +145,224 @@ describe("sandbox managed runtime", () => {
|
||||
await expect(readFile(path.join(localWorkspaceDir, ".paperclip-runtime", "state.json"), "utf8")).resolves.toBe("{}\n");
|
||||
});
|
||||
|
||||
it("syncs git-backed workspaces through a shallow standalone clone and keeps .git out of archives", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-git-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const sourceRepoDir = path.join(rootDir, "source-repo");
|
||||
const localWorkspaceDir = path.join(rootDir, "local-worktree");
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
|
||||
await mkdir(sourceRepoDir, { recursive: true });
|
||||
await git(sourceRepoDir, ["init"]);
|
||||
await git(sourceRepoDir, ["checkout", "-b", "main"]);
|
||||
await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]);
|
||||
await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]);
|
||||
await writeFile(path.join(sourceRepoDir, ".gitignore"), "node_modules/\n", "utf8");
|
||||
await writeFile(path.join(sourceRepoDir, "tracked.txt"), "base\n", "utf8");
|
||||
await writeFile(path.join(sourceRepoDir, "clean.txt"), "from git\n", "utf8");
|
||||
await writeFile(path.join(sourceRepoDir, "deleted.txt"), "delete me\n", "utf8");
|
||||
await git(sourceRepoDir, ["add", ".gitignore", "tracked.txt", "clean.txt", "deleted.txt"]);
|
||||
await git(sourceRepoDir, ["commit", "-m", "base"]);
|
||||
await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]);
|
||||
|
||||
expect((await lstat(path.join(localWorkspaceDir, ".git"))).isFile()).toBe(true);
|
||||
await mkdir(path.join(localWorkspaceDir, "node_modules"), { recursive: true });
|
||||
await writeFile(path.join(localWorkspaceDir, "tracked.txt"), "dirty local\n", "utf8");
|
||||
await writeFile(path.join(localWorkspaceDir, "untracked.txt"), "from local\n", "utf8");
|
||||
await writeFile(path.join(localWorkspaceDir, "node_modules", "cache.bin"), "do not upload\n", "utf8");
|
||||
await rm(path.join(localWorkspaceDir, "deleted.txt"));
|
||||
|
||||
const uploadedTars: { remotePath: string; bytes: Buffer }[] = [];
|
||||
const downloadedTars: { remotePath: string; bytes: Buffer }[] = [];
|
||||
const client: SandboxManagedRuntimeClient = {
|
||||
makeDir: async (remotePath) => {
|
||||
await mkdir(remotePath, { recursive: true });
|
||||
},
|
||||
writeFile: async (remotePath, bytes) => {
|
||||
await mkdir(path.dirname(remotePath), { recursive: true });
|
||||
const buffer = Buffer.from(bytes);
|
||||
if (remotePath.endsWith("-upload.tar")) uploadedTars.push({ remotePath, bytes: buffer });
|
||||
await writeFile(remotePath, buffer);
|
||||
},
|
||||
readFile: async (remotePath) => {
|
||||
const buffer = await readFile(remotePath);
|
||||
if (remotePath.endsWith("workspace-download.tar")) downloadedTars.push({ remotePath, bytes: buffer });
|
||||
return buffer;
|
||||
},
|
||||
listFiles: async () => [],
|
||||
remove: async (remotePath) => {
|
||||
await rm(remotePath, { recursive: true, force: true });
|
||||
},
|
||||
run: async (command) => {
|
||||
await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 });
|
||||
},
|
||||
};
|
||||
|
||||
const prepared = await prepareSandboxManagedRuntime({
|
||||
spec: {
|
||||
transport: "sandbox",
|
||||
provider: "test",
|
||||
sandboxId: "sandbox-1",
|
||||
remoteCwd: remoteWorkspaceDir,
|
||||
timeoutMs: 30_000,
|
||||
apiKey: null,
|
||||
},
|
||||
adapterKey: "test-adapter",
|
||||
client,
|
||||
workspaceLocalDir: localWorkspaceDir,
|
||||
});
|
||||
|
||||
expect((await lstat(path.join(remoteWorkspaceDir, ".git"))).isDirectory()).toBe(true);
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, ".git", "shallow"), "utf8")).resolves.toContain(
|
||||
await git(localWorkspaceDir, ["rev-parse", "HEAD"]),
|
||||
);
|
||||
expect(await git(remoteWorkspaceDir, ["rev-list", "--count", "HEAD"])).toBe("1");
|
||||
expect(await git(remoteWorkspaceDir, ["status", "--short"])).toContain("M tracked.txt");
|
||||
expect(await git(remoteWorkspaceDir, ["status", "--short"])).toContain("?? untracked.txt");
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, "deleted.txt"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||
|
||||
const gitUpload = uploadedTars.find((entry) => path.posix.basename(entry.remotePath) === "git-workspace-upload.tar");
|
||||
const workspaceUpload = uploadedTars.find((entry) => path.posix.basename(entry.remotePath) === "workspace-upload.tar");
|
||||
expect(gitUpload).toBeDefined();
|
||||
expect(workspaceUpload).toBeDefined();
|
||||
const gitMembers = await listTarMembers(rootDir, "git-upload-list.tar", gitUpload!.bytes);
|
||||
const workspaceMembers = await listTarMembers(rootDir, "workspace-upload-list.tar", workspaceUpload!.bytes);
|
||||
expect(gitMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(true);
|
||||
expect(workspaceMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(false);
|
||||
expect(workspaceMembers).toContain("tracked.txt");
|
||||
expect(workspaceMembers).toContain("untracked.txt");
|
||||
expect(workspaceMembers).not.toContain("clean.txt");
|
||||
expect(workspaceMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
|
||||
|
||||
await git(remoteWorkspaceDir, ["config", "user.name", "Paperclip Sandbox"]);
|
||||
await git(remoteWorkspaceDir, ["config", "user.email", "sandbox@paperclip.dev"]);
|
||||
await git(remoteWorkspaceDir, ["add", "-A"]);
|
||||
await git(remoteWorkspaceDir, ["commit", "-m", "sandbox update"]);
|
||||
await writeFile(path.join(remoteWorkspaceDir, "tracked.txt"), "remote dirty\n", "utf8");
|
||||
await writeFile(path.join(remoteWorkspaceDir, "remote-only.txt"), "from sandbox\n", "utf8");
|
||||
|
||||
await prepared.restoreWorkspace();
|
||||
|
||||
expect((await lstat(path.join(localWorkspaceDir, ".git"))).isFile()).toBe(true);
|
||||
expect(await git(localWorkspaceDir, ["log", "-1", "--pretty=%s"])).toBe("sandbox update");
|
||||
await expect(readFile(path.join(localWorkspaceDir, "tracked.txt"), "utf8")).resolves.toBe("remote dirty\n");
|
||||
await expect(readFile(path.join(localWorkspaceDir, "remote-only.txt"), "utf8")).resolves.toBe("from sandbox\n");
|
||||
await expect(readFile(path.join(localWorkspaceDir, "deleted.txt"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(readFile(path.join(localWorkspaceDir, "node_modules", "cache.bin"), "utf8")).resolves.toBe("do not upload\n");
|
||||
|
||||
expect(downloadedTars).toHaveLength(1);
|
||||
const downloadMembers = await listTarMembers(rootDir, "workspace-download-list.tar", downloadedTars[0]!.bytes);
|
||||
expect(downloadMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(false);
|
||||
expect(downloadMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
|
||||
});
|
||||
|
||||
it("excludes unignored dependency trees from git-backed workspace overlay archives", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-unignored-deps-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
const sourceRepoDir = path.join(rootDir, "source-repo");
|
||||
const localWorkspaceDir = path.join(rootDir, "local-worktree");
|
||||
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||
|
||||
await mkdir(sourceRepoDir, { recursive: true });
|
||||
await git(sourceRepoDir, ["init"]);
|
||||
await git(sourceRepoDir, ["checkout", "-b", "main"]);
|
||||
await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]);
|
||||
await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]);
|
||||
await mkdir(path.join(sourceRepoDir, "src"), { recursive: true });
|
||||
await writeFile(path.join(sourceRepoDir, "src", "tracked.ts"), "export const tracked = true;\n", "utf8");
|
||||
await git(sourceRepoDir, ["add", "src/tracked.ts"]);
|
||||
await git(sourceRepoDir, ["commit", "-m", "base"]);
|
||||
await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]);
|
||||
|
||||
await mkdir(path.join(localWorkspaceDir, "node_modules", "root-package"), { recursive: true });
|
||||
await mkdir(path.join(localWorkspaceDir, "packages", "ui", "node_modules", "nested-package"), { recursive: true });
|
||||
await writeFile(path.join(localWorkspaceDir, "node_modules", "root-package", "cache.bin"), "root dependency\n", "utf8");
|
||||
await writeFile(
|
||||
path.join(localWorkspaceDir, "packages", "ui", "node_modules", "nested-package", "cache.bin"),
|
||||
"nested dependency\n",
|
||||
"utf8",
|
||||
);
|
||||
await writeFile(path.join(localWorkspaceDir, "src", "local-only.ts"), "export const local = true;\n", "utf8");
|
||||
|
||||
const uploadedTars: { remotePath: string; bytes: Buffer }[] = [];
|
||||
const downloadedTars: { remotePath: string; bytes: Buffer }[] = [];
|
||||
const client: SandboxManagedRuntimeClient = {
|
||||
makeDir: async (remotePath) => {
|
||||
await mkdir(remotePath, { recursive: true });
|
||||
},
|
||||
writeFile: async (remotePath, bytes) => {
|
||||
await mkdir(path.dirname(remotePath), { recursive: true });
|
||||
const buffer = Buffer.from(bytes);
|
||||
if (remotePath.endsWith("-upload.tar")) uploadedTars.push({ remotePath, bytes: buffer });
|
||||
await writeFile(remotePath, buffer);
|
||||
},
|
||||
readFile: async (remotePath) => {
|
||||
const buffer = await readFile(remotePath);
|
||||
if (remotePath.endsWith("workspace-download.tar")) downloadedTars.push({ remotePath, bytes: buffer });
|
||||
return buffer;
|
||||
},
|
||||
listFiles: async () => [],
|
||||
remove: async (remotePath) => {
|
||||
await rm(remotePath, { recursive: true, force: true });
|
||||
},
|
||||
run: async (command) => {
|
||||
await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 });
|
||||
},
|
||||
};
|
||||
|
||||
const prepared = await prepareSandboxManagedRuntime({
|
||||
spec: {
|
||||
transport: "sandbox",
|
||||
provider: "test",
|
||||
sandboxId: "sandbox-1",
|
||||
remoteCwd: remoteWorkspaceDir,
|
||||
timeoutMs: 30_000,
|
||||
apiKey: null,
|
||||
},
|
||||
adapterKey: "test-adapter",
|
||||
client,
|
||||
workspaceLocalDir: localWorkspaceDir,
|
||||
});
|
||||
|
||||
const workspaceUpload = uploadedTars.find((entry) => path.posix.basename(entry.remotePath) === "workspace-upload.tar");
|
||||
expect(workspaceUpload).toBeDefined();
|
||||
const workspaceMembers = await listTarMembers(rootDir, "unignored-deps-workspace-upload.tar", workspaceUpload!.bytes);
|
||||
expect(workspaceMembers).toContain("src/local-only.ts");
|
||||
expect(workspaceMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
|
||||
expect(workspaceMembers.some((entry) => entry.includes("/node_modules/") || entry.endsWith("/node_modules"))).toBe(false);
|
||||
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, "src", "local-only.ts"), "utf8")).resolves.toBe("export const local = true;\n");
|
||||
await expect(readFile(path.join(remoteWorkspaceDir, "node_modules", "root-package", "cache.bin"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||
await expect(
|
||||
readFile(path.join(remoteWorkspaceDir, "packages", "ui", "node_modules", "nested-package", "cache.bin"), "utf8"),
|
||||
).rejects.toMatchObject({ code: "ENOENT" });
|
||||
|
||||
await mkdir(path.join(remoteWorkspaceDir, "node_modules", "sandbox-package"), { recursive: true });
|
||||
await mkdir(path.join(remoteWorkspaceDir, "packages", "ui", "node_modules", "sandbox-package"), { recursive: true });
|
||||
await writeFile(path.join(remoteWorkspaceDir, "node_modules", "sandbox-package", "cache.bin"), "sandbox root dependency\n", "utf8");
|
||||
await writeFile(
|
||||
path.join(remoteWorkspaceDir, "packages", "ui", "node_modules", "sandbox-package", "cache.bin"),
|
||||
"sandbox nested dependency\n",
|
||||
"utf8",
|
||||
);
|
||||
await writeFile(path.join(remoteWorkspaceDir, "src", "remote-only.ts"), "export const remote = true;\n", "utf8");
|
||||
|
||||
await prepared.restoreWorkspace();
|
||||
|
||||
await expect(readFile(path.join(localWorkspaceDir, "node_modules", "root-package", "cache.bin"), "utf8")).resolves.toBe("root dependency\n");
|
||||
await expect(
|
||||
readFile(path.join(localWorkspaceDir, "packages", "ui", "node_modules", "nested-package", "cache.bin"), "utf8"),
|
||||
).resolves.toBe("nested dependency\n");
|
||||
await expect(readFile(path.join(localWorkspaceDir, "src", "remote-only.ts"), "utf8")).resolves.toBe("export const remote = true;\n");
|
||||
await expect(readFile(path.join(localWorkspaceDir, "node_modules", "sandbox-package", "cache.bin"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||
|
||||
expect(downloadedTars).toHaveLength(1);
|
||||
const downloadMembers = await listTarMembers(rootDir, "unignored-deps-workspace-download.tar", downloadedTars[0]!.bytes);
|
||||
expect(downloadMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(false);
|
||||
expect(downloadMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
|
||||
expect(downloadMembers.some((entry) => entry.includes("/node_modules/") || entry.endsWith("/node_modules"))).toBe(false);
|
||||
});
|
||||
|
||||
it("builds workspace/asset tarballs without a './' self-entry (so untar does not chmod/utime an unowned target dir)", async () => {
|
||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-tarself-"));
|
||||
cleanupDirs.push(rootDir);
|
||||
|
||||
@@ -3,10 +3,44 @@ import { constants as fsConstants, promises as fs } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import {
|
||||
buildRemoteGitDeltaBundleScript,
|
||||
createImportedGitRef,
|
||||
createRemoteGitExportRef,
|
||||
deleteLocalGitRef,
|
||||
fetchGitBundleIntoLocalRef,
|
||||
GIT_ARCHIVE_EXCLUDES,
|
||||
integrateImportedGitHead,
|
||||
readGitWorkspaceSnapshot,
|
||||
withShallowGitWorkspaceClone,
|
||||
} from "./git-workspace-sync.js";
|
||||
import { captureDirectorySnapshot, mergeDirectoryWithBaseline } from "./workspace-restore-merge.js";
|
||||
import { createRuntimeProgressReporter, type RuntimeProgressSink } from "./runtime-progress.js";
|
||||
import {
|
||||
createRuntimeProgressReporter,
|
||||
type RuntimeProgressDirection,
|
||||
type RuntimeProgressPhase,
|
||||
type RuntimeProgressSink,
|
||||
} from "./runtime-progress.js";
|
||||
import { isRelativePathOrDescendant, shouldExcludePath } from "./exclude-patterns.js";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
const SANDBOX_WORKSPACE_HEAVY_DIR_NAMES = [
|
||||
"node_modules",
|
||||
"vendor",
|
||||
"dist",
|
||||
"build",
|
||||
"out",
|
||||
"coverage",
|
||||
".next",
|
||||
".turbo",
|
||||
".cache",
|
||||
] as const;
|
||||
const SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES = SANDBOX_WORKSPACE_HEAVY_DIR_NAMES.flatMap((entry) => [
|
||||
entry,
|
||||
`${entry}/*`,
|
||||
`*/${entry}`,
|
||||
`*/${entry}/*`,
|
||||
]);
|
||||
|
||||
export interface SandboxRemoteExecutionSpec {
|
||||
transport: "sandbox";
|
||||
@@ -208,8 +242,28 @@ async function walkDirectory(root: string, relative = ""): Promise<string[]> {
|
||||
return out.sort((left, right) => right.length - left.length);
|
||||
}
|
||||
|
||||
function isRelativePathOrDescendant(relative: string, candidate: string): boolean {
|
||||
return relative === candidate || relative.startsWith(`${candidate}/`);
|
||||
async function copyWorkspaceEntry(sourceRoot: string, targetRoot: string, relative: string): Promise<void> {
|
||||
const sourcePath = path.join(sourceRoot, relative);
|
||||
const targetPath = path.join(targetRoot, relative);
|
||||
const stats = await fs.lstat(sourcePath);
|
||||
|
||||
if (stats.isDirectory()) {
|
||||
await fs.mkdir(targetPath, { recursive: true });
|
||||
return;
|
||||
}
|
||||
|
||||
await fs.mkdir(path.dirname(targetPath), { recursive: true });
|
||||
await fs.rm(targetPath, { recursive: true, force: true }).catch(() => undefined);
|
||||
if (stats.isSymbolicLink()) {
|
||||
const linkTarget = await fs.readlink(sourcePath);
|
||||
await fs.symlink(linkTarget, targetPath);
|
||||
return;
|
||||
}
|
||||
|
||||
await fs.copyFile(sourcePath, targetPath, fsConstants.COPYFILE_FICLONE).catch(async () => {
|
||||
await fs.copyFile(sourcePath, targetPath);
|
||||
});
|
||||
await fs.chmod(targetPath, stats.mode);
|
||||
}
|
||||
|
||||
export async function mirrorDirectory(
|
||||
@@ -231,33 +285,24 @@ export async function mirrorDirectory(
|
||||
}
|
||||
}
|
||||
|
||||
const copyEntry = async (relative: string) => {
|
||||
const sourcePath = path.join(sourceDir, relative);
|
||||
const targetPath = path.join(targetDir, relative);
|
||||
const stats = await fs.lstat(sourcePath);
|
||||
|
||||
if (stats.isDirectory()) {
|
||||
await fs.mkdir(targetPath, { recursive: true });
|
||||
return;
|
||||
}
|
||||
|
||||
await fs.mkdir(path.dirname(targetPath), { recursive: true });
|
||||
await fs.rm(targetPath, { recursive: true, force: true }).catch(() => undefined);
|
||||
if (stats.isSymbolicLink()) {
|
||||
const linkTarget = await fs.readlink(sourcePath);
|
||||
await fs.symlink(linkTarget, targetPath);
|
||||
return;
|
||||
}
|
||||
|
||||
await fs.copyFile(sourcePath, targetPath, fsConstants.COPYFILE_FICLONE).catch(async () => {
|
||||
await fs.copyFile(sourcePath, targetPath);
|
||||
});
|
||||
await fs.chmod(targetPath, stats.mode);
|
||||
};
|
||||
|
||||
const entries = (await walkDirectory(sourceDir)).sort((left, right) => left.localeCompare(right));
|
||||
for (const relative of entries) {
|
||||
await copyEntry(relative);
|
||||
await copyWorkspaceEntry(sourceDir, targetDir, relative);
|
||||
}
|
||||
}
|
||||
|
||||
async function copySelectedWorkspaceEntries(input: {
|
||||
sourceDir: string;
|
||||
targetDir: string;
|
||||
relativePaths: string[];
|
||||
exclude: string[];
|
||||
}): Promise<void> {
|
||||
await fs.mkdir(input.targetDir, { recursive: true });
|
||||
for (const relative of input.relativePaths) {
|
||||
if (shouldExcludePath(relative, input.exclude)) continue;
|
||||
const sourceStats = await fs.lstat(path.join(input.sourceDir, relative)).catch(() => null);
|
||||
if (!sourceStats) continue;
|
||||
await copyWorkspaceEntry(input.sourceDir, input.targetDir, relative);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -275,15 +320,37 @@ function tarExcludeFlags(exclude: string[] | undefined): string {
|
||||
return ["._*", ...(exclude ?? [])].map((entry) => `--exclude ${shellQuote(entry)}`).join(" ");
|
||||
}
|
||||
|
||||
function mergeExcludes(...groups: Array<string[] | undefined>): string[] {
|
||||
return [...new Set(groups.flatMap((group) => group ?? []))];
|
||||
}
|
||||
|
||||
function preserveFindArgs(entries: string[]): string {
|
||||
return entries.map((entry) => `! -name ${shellQuote(entry)}`).join(" ");
|
||||
}
|
||||
|
||||
async function removeDeletedPathsInSandbox(input: {
|
||||
client: SandboxManagedRuntimeClient;
|
||||
spec: SandboxRemoteExecutionSpec;
|
||||
remoteDir: string;
|
||||
deletedPaths: string[];
|
||||
}): Promise<void> {
|
||||
if (input.deletedPaths.length === 0) return;
|
||||
const quotedPaths = input.deletedPaths.map((entry) => shellQuote(entry)).join(" ");
|
||||
await input.client.run(
|
||||
`sh -c ${shellQuote(`cd ${shellQuote(input.remoteDir)} && rm -rf -- ${quotedPaths}`)}`,
|
||||
{ timeoutMs: input.spec.timeoutMs },
|
||||
);
|
||||
}
|
||||
|
||||
// Bridge a single byte-level transfer to the throttled progress reporter. The
|
||||
// transport reports decoded bytes via `options.onProgress`; the reporter turns
|
||||
// them into a throttled, fully-formatted log line. `finish()` emits the terminal
|
||||
// completion line (idempotent) once the transfer returns.
|
||||
function makeTransferProgress(
|
||||
sink: RuntimeProgressSink | undefined,
|
||||
phase: "Syncing" | "Restoring",
|
||||
direction: "to" | "from",
|
||||
label: string,
|
||||
phase: RuntimeProgressPhase,
|
||||
direction: RuntimeProgressDirection,
|
||||
label?: string,
|
||||
): { options: SandboxTransferProgressOptions | undefined; finish: () => Promise<void> } {
|
||||
if (!sink) return { options: undefined, finish: async () => {} };
|
||||
const reporter = createRuntimeProgressReporter({
|
||||
@@ -320,16 +387,75 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
}): Promise<PreparedSandboxManagedRuntime> {
|
||||
const workspaceRemoteDir = input.workspaceRemoteDir ?? input.spec.remoteCwd;
|
||||
const runtimeRootDir = path.posix.join(workspaceRemoteDir, ".paperclip-runtime", input.adapterKey);
|
||||
const gitSnapshot = await readGitWorkspaceSnapshot(input.workspaceLocalDir);
|
||||
const gitIgnoredExcludes = gitSnapshot?.ignoredPaths;
|
||||
const workspaceArchiveExclude = mergeExcludes(
|
||||
SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES,
|
||||
[...GIT_ARCHIVE_EXCLUDES],
|
||||
input.workspaceExclude,
|
||||
gitIgnoredExcludes,
|
||||
);
|
||||
const restoreExclude = mergeExcludes(
|
||||
SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES,
|
||||
[...GIT_ARCHIVE_EXCLUDES],
|
||||
[".paperclip-runtime"],
|
||||
input.preserveAbsentOnRestore,
|
||||
input.workspaceExclude,
|
||||
gitIgnoredExcludes,
|
||||
);
|
||||
const baselineSnapshot = await captureDirectorySnapshot(input.workspaceLocalDir, {
|
||||
exclude: [...new Set([".paperclip-runtime", ...(input.preserveAbsentOnRestore ?? []), ...(input.workspaceExclude ?? [])])],
|
||||
exclude: restoreExclude,
|
||||
});
|
||||
|
||||
await withTempDir("paperclip-sandbox-sync-", async (tempDir) => {
|
||||
const preservedNames = new Set([
|
||||
".paperclip-runtime",
|
||||
...(gitSnapshot ? [".git"] : []),
|
||||
...(input.preserveAbsentOnRestore ?? []),
|
||||
]);
|
||||
if (gitSnapshot) {
|
||||
await withShallowGitWorkspaceClone({
|
||||
localDir: input.workspaceLocalDir,
|
||||
snapshot: gitSnapshot,
|
||||
}, async (cloneDir) => {
|
||||
const gitTarPath = path.join(tempDir, "git-workspace.tar");
|
||||
await createTarballFromDirectory({
|
||||
localDir: cloneDir,
|
||||
archivePath: gitTarPath,
|
||||
exclude: [".paperclip-runtime"],
|
||||
});
|
||||
const gitTarBytes = await fs.readFile(gitTarPath);
|
||||
const remoteGitTar = path.posix.join(runtimeRootDir, "git-workspace-upload.tar");
|
||||
await input.client.makeDir(runtimeRootDir);
|
||||
const gitUpload = makeTransferProgress(input.onProgress, "Syncing", "to", "git history");
|
||||
await input.client.writeFile(remoteGitTar, toArrayBuffer(gitTarBytes), gitUpload.options);
|
||||
await gitUpload.finish();
|
||||
await input.client.run(
|
||||
`sh -c ${shellQuote(
|
||||
`mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
|
||||
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${preserveFindArgs([".paperclip-runtime"])} -exec rm -rf -- {} + && ` +
|
||||
`tar -xf ${shellQuote(remoteGitTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
|
||||
`rm -f ${shellQuote(remoteGitTar)}`,
|
||||
)}`,
|
||||
{ timeoutMs: input.spec.timeoutMs },
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
const workspaceTarPath = path.join(tempDir, "workspace.tar");
|
||||
const workspaceArchiveDir = gitSnapshot ? path.join(tempDir, "workspace-overlay") : input.workspaceLocalDir;
|
||||
if (gitSnapshot) {
|
||||
await copySelectedWorkspaceEntries({
|
||||
sourceDir: input.workspaceLocalDir,
|
||||
targetDir: workspaceArchiveDir,
|
||||
relativePaths: gitSnapshot.overlayPaths,
|
||||
exclude: workspaceArchiveExclude,
|
||||
});
|
||||
}
|
||||
await createTarballFromDirectory({
|
||||
localDir: input.workspaceLocalDir,
|
||||
localDir: workspaceArchiveDir,
|
||||
archivePath: workspaceTarPath,
|
||||
exclude: input.workspaceExclude,
|
||||
exclude: gitSnapshot ? undefined : workspaceArchiveExclude,
|
||||
});
|
||||
const workspaceTarBytes = await fs.readFile(workspaceTarPath);
|
||||
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-upload.tar");
|
||||
@@ -341,17 +467,26 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
workspaceUpload.options,
|
||||
);
|
||||
await workspaceUpload.finish();
|
||||
const preservedNames = new Set([".paperclip-runtime", ...(input.preserveAbsentOnRestore ?? [])]);
|
||||
const findPreserveArgs = [...preservedNames].map((entry) => `! -name ${shellQuote(entry)}`).join(" ");
|
||||
const extractWorkspaceTarCommand = gitSnapshot
|
||||
? `mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
|
||||
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
|
||||
`rm -f ${shellQuote(remoteWorkspaceTar)}`
|
||||
: `mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
|
||||
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${preserveFindArgs([...preservedNames])} -exec rm -rf -- {} + && ` +
|
||||
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
|
||||
`rm -f ${shellQuote(remoteWorkspaceTar)}`;
|
||||
await input.client.run(
|
||||
`sh -c ${shellQuote(
|
||||
`mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
|
||||
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${findPreserveArgs} -exec rm -rf -- {} + && ` +
|
||||
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
|
||||
`rm -f ${shellQuote(remoteWorkspaceTar)}`,
|
||||
)}`,
|
||||
`sh -c ${shellQuote(extractWorkspaceTarCommand)}`,
|
||||
{ timeoutMs: input.spec.timeoutMs },
|
||||
);
|
||||
if (gitSnapshot) {
|
||||
await removeDeletedPathsInSandbox({
|
||||
client: input.client,
|
||||
spec: input.spec,
|
||||
remoteDir: workspaceRemoteDir,
|
||||
deletedPaths: gitSnapshot.deletedPaths,
|
||||
});
|
||||
}
|
||||
|
||||
for (const asset of input.assets ?? []) {
|
||||
const assetTarPath = path.join(tempDir, `${asset.key}.tar`);
|
||||
@@ -392,31 +527,76 @@ export async function prepareSandboxManagedRuntime(input: {
|
||||
restoreWorkspace: async (onProgress?: RuntimeProgressSink) => {
|
||||
const restoreSink = onProgress ?? input.onProgress;
|
||||
await withTempDir("paperclip-sandbox-restore-", async (tempDir) => {
|
||||
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-download.tar");
|
||||
await input.client.run(
|
||||
`sh -c ${shellQuote(
|
||||
`mkdir -p ${shellQuote(runtimeRootDir)} && ` +
|
||||
`tar -cf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} ` +
|
||||
`${tarExcludeFlags(input.workspaceExclude)} .`,
|
||||
)}`,
|
||||
{ timeoutMs: input.spec.timeoutMs },
|
||||
);
|
||||
const workspaceRestore = makeTransferProgress(restoreSink, "Restoring", "from", "workspace");
|
||||
const archiveBytes = await input.client.readFile(remoteWorkspaceTar, workspaceRestore.options);
|
||||
await workspaceRestore.finish();
|
||||
await input.client.remove(remoteWorkspaceTar).catch(() => undefined);
|
||||
const localArchivePath = path.join(tempDir, "workspace.tar");
|
||||
const extractedDir = path.join(tempDir, "workspace");
|
||||
await fs.writeFile(localArchivePath, toBuffer(archiveBytes));
|
||||
await extractTarballToDirectory({
|
||||
archivePath: localArchivePath,
|
||||
localDir: extractedDir,
|
||||
});
|
||||
await mergeDirectoryWithBaseline({
|
||||
baseline: baselineSnapshot,
|
||||
sourceDir: extractedDir,
|
||||
targetDir: input.workspaceLocalDir,
|
||||
});
|
||||
let importedRef: string | null = null;
|
||||
let importedHead: string | null = null;
|
||||
try {
|
||||
if (gitSnapshot) {
|
||||
importedRef = createImportedGitRef("sandbox");
|
||||
const remoteGitBundle = path.posix.join(runtimeRootDir, "git-delta.bundle");
|
||||
const exportRef = createRemoteGitExportRef("sandbox");
|
||||
await input.client.run(
|
||||
`sh -c ${shellQuote(buildRemoteGitDeltaBundleScript({
|
||||
remoteDir: workspaceRemoteDir,
|
||||
baseSha: gitSnapshot.headCommit,
|
||||
exportRef,
|
||||
bundlePath: remoteGitBundle,
|
||||
}))}`,
|
||||
{ timeoutMs: input.spec.timeoutMs },
|
||||
);
|
||||
const gitExport = makeTransferProgress(restoreSink, "Exporting git history", "from");
|
||||
const bundleBytes = await input.client.readFile(remoteGitBundle, gitExport.options);
|
||||
await gitExport.finish();
|
||||
await input.client.remove(remoteGitBundle).catch(() => undefined);
|
||||
const bundlePath = path.join(tempDir, "git-delta.bundle");
|
||||
await fs.writeFile(bundlePath, toBuffer(bundleBytes));
|
||||
importedHead = await fetchGitBundleIntoLocalRef({
|
||||
localDir: input.workspaceLocalDir,
|
||||
bundlePath,
|
||||
exportRef,
|
||||
importedRef,
|
||||
baseSha: gitSnapshot.headCommit,
|
||||
});
|
||||
}
|
||||
|
||||
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-download.tar");
|
||||
await input.client.run(
|
||||
`sh -c ${shellQuote(
|
||||
`mkdir -p ${shellQuote(runtimeRootDir)} && ` +
|
||||
`tar -cf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} ` +
|
||||
`${tarExcludeFlags(restoreExclude)} .`,
|
||||
)}`,
|
||||
{ timeoutMs: input.spec.timeoutMs },
|
||||
);
|
||||
const workspaceRestore = makeTransferProgress(restoreSink, "Restoring", "from", "workspace");
|
||||
const archiveBytes = await input.client.readFile(remoteWorkspaceTar, workspaceRestore.options);
|
||||
await workspaceRestore.finish();
|
||||
await input.client.remove(remoteWorkspaceTar).catch(() => undefined);
|
||||
const localArchivePath = path.join(tempDir, "workspace.tar");
|
||||
const extractedDir = path.join(tempDir, "workspace");
|
||||
await fs.writeFile(localArchivePath, toBuffer(archiveBytes));
|
||||
await extractTarballToDirectory({
|
||||
archivePath: localArchivePath,
|
||||
localDir: extractedDir,
|
||||
});
|
||||
const gitHeadToIntegrate = importedHead;
|
||||
await mergeDirectoryWithBaseline({
|
||||
baseline: baselineSnapshot,
|
||||
sourceDir: extractedDir,
|
||||
targetDir: input.workspaceLocalDir,
|
||||
beforeApply: gitHeadToIntegrate
|
||||
? async () => {
|
||||
await integrateImportedGitHead({
|
||||
localDir: input.workspaceLocalDir,
|
||||
importedHead: gitHeadToIntegrate,
|
||||
});
|
||||
}
|
||||
: undefined,
|
||||
});
|
||||
} finally {
|
||||
if (importedRef) {
|
||||
await deleteLocalGitRef({ localDir: input.workspaceLocalDir, ref: importedRef });
|
||||
}
|
||||
}
|
||||
});
|
||||
},
|
||||
};
|
||||
|
||||
@@ -2,6 +2,7 @@ import { createHash } from "node:crypto";
|
||||
import { createReadStream } from "node:fs";
|
||||
import { constants as fsConstants, promises as fs } from "node:fs";
|
||||
import path from "node:path";
|
||||
import { shouldExcludePath } from "./exclude-patterns.js";
|
||||
|
||||
type SnapshotEntry =
|
||||
| { kind: "dir" }
|
||||
@@ -13,14 +14,6 @@ export interface DirectorySnapshot {
|
||||
entries: Map<string, SnapshotEntry>;
|
||||
}
|
||||
|
||||
function isRelativePathOrDescendant(relative: string, candidate: string): boolean {
|
||||
return relative === candidate || relative.startsWith(`${candidate}/`);
|
||||
}
|
||||
|
||||
function shouldExclude(relative: string, exclude: readonly string[]): boolean {
|
||||
return exclude.some((candidate) => isRelativePathOrDescendant(relative, candidate));
|
||||
}
|
||||
|
||||
async function hashFile(filePath: string): Promise<string> {
|
||||
return await new Promise((resolve, reject) => {
|
||||
const hash = createHash("sha256");
|
||||
@@ -43,7 +36,7 @@ async function walkDirectory(
|
||||
|
||||
for (const entry of entries) {
|
||||
const nextRelative = relative ? path.posix.join(relative, entry.name) : entry.name;
|
||||
if (shouldExclude(nextRelative, exclude)) continue;
|
||||
if (shouldExcludePath(nextRelative, exclude)) continue;
|
||||
|
||||
const fullPath = path.join(root, nextRelative);
|
||||
const stats = await fs.lstat(fullPath);
|
||||
|
||||
@@ -313,6 +313,82 @@ describe("resolveExecutionRunAdapterConfig", () => {
|
||||
details: { code: "low_trust_inline_sensitive_env_denied" },
|
||||
});
|
||||
});
|
||||
|
||||
it("fails push-capability preflight when no GitHub write credential is bound at agent or project scope", async () => {
|
||||
await expect(resolveExecutionRunAdapterConfig({
|
||||
companyId: "company-1",
|
||||
agentId: "agent-1",
|
||||
issueId: "issue-1",
|
||||
executionRunConfig: { env: { AGENT_ONLY: "agent-only" } },
|
||||
projectEnv: { PROJECT_ONLY: "project-only" },
|
||||
requiredScopedEnvBinding: {
|
||||
keys: ["GH_TOKEN", "GITHUB_TOKEN"],
|
||||
consumerScopes: ["agent", "project"],
|
||||
reason: "push_write_credential_missing",
|
||||
remediation: "GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN bound at project or agent scope.",
|
||||
},
|
||||
secretsSvc: {
|
||||
resolveAdapterConfigForRuntime: vi.fn(),
|
||||
resolveEnvBindings: vi.fn(),
|
||||
} as any,
|
||||
})).rejects.toMatchObject({
|
||||
code: "configuration_incomplete",
|
||||
message: expect.stringContaining("GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN"),
|
||||
resultJson: {
|
||||
configurationIncomplete: {
|
||||
reason: "push_write_credential_missing",
|
||||
requiredEnvKeys: ["GH_TOKEN", "GITHUB_TOKEN"],
|
||||
requiredScopes: ["agent", "project"],
|
||||
missingBindings: [],
|
||||
},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("passes push-capability preflight when a project-scoped GitHub credential is configured", async () => {
|
||||
const resolveAdapterConfigForRuntime = vi.fn().mockResolvedValue({
|
||||
config: { env: { AGENT_ONLY: "agent-only" } },
|
||||
secretKeys: new Set<string>(),
|
||||
manifest: [],
|
||||
});
|
||||
const resolveEnvBindings = vi.fn().mockResolvedValue({
|
||||
env: { GH_TOKEN: "github-token" },
|
||||
secretKeys: new Set(["GH_TOKEN"]),
|
||||
manifest: [],
|
||||
});
|
||||
const collectMissingRuntimeBindings = vi.fn().mockResolvedValue([]);
|
||||
|
||||
const result = await resolveExecutionRunAdapterConfig({
|
||||
companyId: "company-1",
|
||||
agentId: "agent-1",
|
||||
issueId: "issue-1",
|
||||
projectId: "project-1",
|
||||
executionRunConfig: { env: { AGENT_ONLY: "agent-only" } },
|
||||
projectEnv: { GH_TOKEN: { type: "plain", value: "github-token" } },
|
||||
requiredScopedEnvBinding: {
|
||||
keys: ["GH_TOKEN", "GITHUB_TOKEN"],
|
||||
consumerScopes: ["agent", "project"],
|
||||
reason: "push_write_credential_missing",
|
||||
remediation: "GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN bound at project or agent scope.",
|
||||
},
|
||||
secretsSvc: {
|
||||
resolveAdapterConfigForRuntime,
|
||||
resolveEnvBindings,
|
||||
collectMissingRuntimeBindings,
|
||||
} as any,
|
||||
});
|
||||
|
||||
expect(result.resolvedConfig.env).toEqual({
|
||||
AGENT_ONLY: "agent-only",
|
||||
GH_TOKEN: "github-token",
|
||||
});
|
||||
expect(resolveEnvBindings).toHaveBeenCalledOnce();
|
||||
expect(collectMissingRuntimeBindings).toHaveBeenCalledTimes(2);
|
||||
expect(collectMissingRuntimeBindings.mock.calls[1]?.[2]).toMatchObject({
|
||||
consumerType: "project",
|
||||
consumerId: "project-1",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("extractMentionedSkillIdsFromSources", () => {
|
||||
|
||||
@@ -1,3 +1,8 @@
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { agents } from "@paperclipai/db";
|
||||
import { sessionCodec as codexSessionCodec } from "@paperclipai/adapter-codex-local/server";
|
||||
@@ -5,6 +10,7 @@ import { resolveDefaultAgentWorkspaceDir } from "../home-paths.js";
|
||||
import {
|
||||
applyPersistedExecutionWorkspaceConfig,
|
||||
assertGitSensitiveAdapterWorkspaceValid,
|
||||
assertPushCapabilityCheckoutValid,
|
||||
buildRealizedExecutionWorkspaceFromPersisted,
|
||||
buildExplicitResumeSessionOverride,
|
||||
deriveTaskKeyWithHeartbeatFallback,
|
||||
@@ -16,6 +22,7 @@ import {
|
||||
prioritizeProjectWorkspaceCandidatesForRun,
|
||||
parseSessionCompactionPolicy,
|
||||
resolveNextSessionState,
|
||||
requiresPushCapabilityPreflight,
|
||||
resolveWorkspaceAfterLowTrustPreflight,
|
||||
resolveRuntimeSessionParamsForWorkspace,
|
||||
shouldDeferFollowupWakeForSameIssue,
|
||||
@@ -29,6 +36,8 @@ import {
|
||||
} from "../services/heartbeat.ts";
|
||||
import type { TrustPresetResolution } from "../services/trust-preset-resolver.ts";
|
||||
|
||||
const execFile = promisify(execFileCallback);
|
||||
|
||||
function buildResolvedWorkspace(overrides: Partial<ResolvedWorkspaceForRun> = {}): ResolvedWorkspaceForRun {
|
||||
return {
|
||||
cwd: "/tmp/project",
|
||||
@@ -105,6 +114,19 @@ function buildWorkspaceValidationInput(
|
||||
};
|
||||
}
|
||||
|
||||
async function runGit(cwd: string, args: string[]) {
|
||||
await execFile("git", args, { cwd });
|
||||
}
|
||||
|
||||
async function createGitCheckout(options: { withRemote: boolean }) {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-push-preflight-"));
|
||||
await runGit(root, ["init"]);
|
||||
if (options.withRemote) {
|
||||
await runGit(root, ["remote", "add", "origin", "https://github.com/example/repo.git"]);
|
||||
}
|
||||
return root;
|
||||
}
|
||||
|
||||
async function expectWorkspaceValidationFailure(
|
||||
input: WorkspaceValidationInput,
|
||||
reason: string,
|
||||
@@ -377,6 +399,72 @@ describe("assertGitSensitiveAdapterWorkspaceValid", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("assertPushCapabilityCheckoutValid", () => {
|
||||
it("rejects a GitHub PR workflow checkout without a configured push remote", async () => {
|
||||
const cwd = await createGitCheckout({ withRemote: false });
|
||||
try {
|
||||
await expect(assertPushCapabilityCheckoutValid({
|
||||
enabled: true,
|
||||
issue: {
|
||||
id: "issue-1",
|
||||
identifier: "PAP-1",
|
||||
},
|
||||
cwd,
|
||||
})).rejects.toMatchObject({
|
||||
code: "workspace_validation_failed",
|
||||
message: expect.stringContaining("has no configured push remote"),
|
||||
resultJson: {
|
||||
workspaceValidation: expect.objectContaining({
|
||||
reason: "missing_git_push_remote",
|
||||
issueId: "issue-1",
|
||||
executionWorkspaceCwd: cwd,
|
||||
}),
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
await fs.rm(cwd, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it("allows a GitHub PR workflow checkout when a push remote is configured", async () => {
|
||||
const cwd = await createGitCheckout({ withRemote: true });
|
||||
try {
|
||||
await expect(assertPushCapabilityCheckoutValid({
|
||||
enabled: true,
|
||||
issue: {
|
||||
id: "issue-1",
|
||||
identifier: "PAP-1",
|
||||
},
|
||||
cwd,
|
||||
})).resolves.toBeUndefined();
|
||||
} finally {
|
||||
await fs.rm(cwd, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("requiresPushCapabilityPreflight", () => {
|
||||
it("only enables the guard when the issue explicitly mentions the GitHub PR workflow skill", () => {
|
||||
expect(requiresPushCapabilityPreflight({
|
||||
adapterType: "codex_local",
|
||||
issueId: "issue-1",
|
||||
explicitRunScopedSkillKeys: ["paperclipai/bundled/software-development/github-pr-workflow"],
|
||||
})).toBe(true);
|
||||
|
||||
expect(requiresPushCapabilityPreflight({
|
||||
adapterType: "codex_local",
|
||||
issueId: "issue-1",
|
||||
explicitRunScopedSkillKeys: [],
|
||||
})).toBe(false);
|
||||
|
||||
expect(requiresPushCapabilityPreflight({
|
||||
adapterType: "cursor-cloud",
|
||||
issueId: "issue-1",
|
||||
explicitRunScopedSkillKeys: ["paperclipai/bundled/software-development/github-pr-workflow"],
|
||||
})).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("stripHostWorkspaceProvisionForLowTrustSandbox", () => {
|
||||
it("removes only the host-side provision command for sandbox-backed low-trust runs", () => {
|
||||
const config = {
|
||||
|
||||
@@ -61,12 +61,13 @@ async function waitFor(condition: () => boolean | Promise<boolean>, timeoutMs =
|
||||
throw new Error("Timed out waiting for condition");
|
||||
}
|
||||
|
||||
async function deleteHeartbeatRunsAfterActivityLogDrains(db: Db) {
|
||||
async function deleteHeartbeatRunsAndWakeupsAfterActivityLogDrains(db: Db) {
|
||||
let lastError: unknown = null;
|
||||
for (let attempt = 0; attempt < 10; attempt += 1) {
|
||||
await db.delete(activityLog);
|
||||
try {
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agentWakeupRequests);
|
||||
return;
|
||||
} catch (error) {
|
||||
lastError = error;
|
||||
@@ -527,8 +528,7 @@ describeEmbeddedPostgres("low-trust red-team HTTP route regression suite", () =>
|
||||
await db.delete(issueRelations);
|
||||
await db.delete(activityLog);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await deleteHeartbeatRunsAfterActivityLogDrains(db);
|
||||
await db.delete(agentWakeupRequests);
|
||||
await deleteHeartbeatRunsAndWakeupsAfterActivityLogDrains(db);
|
||||
await db.delete(issues);
|
||||
await db.delete(agentRuntimeState);
|
||||
await db.delete(agents);
|
||||
|
||||
@@ -260,6 +260,9 @@ const WORKSPACE_VALIDATION_FAILURE_CODE = "workspace_validation_failed";
|
||||
const WORKSPACE_VALIDATION_RECOVERY_CAUSE = "workspace_validation_failed";
|
||||
const CONFIGURATION_INCOMPLETE_FAILURE_CODE = "configuration_incomplete";
|
||||
const CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE = "configuration_incomplete";
|
||||
const GITHUB_PR_WORKFLOW_SKILL_KEY = "paperclipai/bundled/software-development/github-pr-workflow";
|
||||
const GITHUB_PR_WORKFLOW_SKILL_SLUG = "github-pr-workflow";
|
||||
const PUSH_CAPABILITY_ENV_KEYS = ["GH_TOKEN", "GITHUB_TOKEN"] as const;
|
||||
// Keep this in sync with local adapters that require a git workspace before launch.
|
||||
const GIT_SENSITIVE_LOCAL_ADAPTER_TYPES = new Set([
|
||||
"acpx_local",
|
||||
@@ -411,6 +414,34 @@ function formatMissingBindingForOperator(missing: MissingRuntimeBinding): string
|
||||
return `secret ${secretLabel} not bound at ${missing.consumerType} ${missing.configPath}`;
|
||||
}
|
||||
|
||||
function isConfiguredEnvBindingValue(binding: unknown) {
|
||||
const parsed = envBindingSchema.safeParse(binding);
|
||||
if (!parsed.success) return false;
|
||||
const value = parsed.data;
|
||||
if (typeof value === "string") return value.trim().length > 0;
|
||||
if (value.type === "plain") return value.value.trim().length > 0;
|
||||
return true;
|
||||
}
|
||||
|
||||
function hasGithubPrWorkflowSkill(desiredSkills: string[]) {
|
||||
return desiredSkills.some((skill) => {
|
||||
const normalized = skill.trim();
|
||||
return normalized === GITHUB_PR_WORKFLOW_SKILL_KEY
|
||||
|| normalized === GITHUB_PR_WORKFLOW_SKILL_SLUG
|
||||
|| normalized.endsWith(`/${GITHUB_PR_WORKFLOW_SKILL_SLUG}`);
|
||||
});
|
||||
}
|
||||
|
||||
export function requiresPushCapabilityPreflight(input: {
|
||||
adapterType: string;
|
||||
issueId: string | null | undefined;
|
||||
explicitRunScopedSkillKeys: string[];
|
||||
}) {
|
||||
return Boolean(input.issueId)
|
||||
&& GIT_SENSITIVE_LOCAL_ADAPTER_TYPES.has(input.adapterType)
|
||||
&& hasGithubPrWorkflowSkill(input.explicitRunScopedSkillKeys);
|
||||
}
|
||||
|
||||
const LOW_TRUST_SENSITIVE_ENV_KEY_RE =
|
||||
/(api[-_]?key|access[-_]?token|auth(?:_?token)?|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring)/i;
|
||||
|
||||
@@ -466,11 +497,18 @@ export async function resolveExecutionRunAdapterConfig(input: {
|
||||
routineEnv?: unknown;
|
||||
secretsSvc: RuntimeConfigSecretResolver;
|
||||
trustPreset?: TrustPresetResolution;
|
||||
requiredScopedEnvBinding?: {
|
||||
keys: string[];
|
||||
consumerScopes: Array<"agent" | "project">;
|
||||
reason: string;
|
||||
remediation: string;
|
||||
};
|
||||
}) {
|
||||
const executionRunConfig = stripPaperclipRuntimeEnvFromAdapterConfig(input.executionRunConfig);
|
||||
const environmentEnv = stripPaperclipRuntimeEnvBindings(input.environmentEnv);
|
||||
const projectEnv = stripPaperclipRuntimeEnvBindings(input.projectEnv);
|
||||
const routineEnv = stripPaperclipRuntimeEnvBindings(input.routineEnv);
|
||||
const agentEnv = parseObject(executionRunConfig.env);
|
||||
const lowTrustAllowedBindingIds = input.trustPreset?.kind === "low_trust_review"
|
||||
? input.trustPreset.boundary.allowedSecretBindingIds ?? []
|
||||
: undefined;
|
||||
@@ -480,6 +518,31 @@ export async function resolveExecutionRunAdapterConfig(input: {
|
||||
assertLowTrustEnvConfigAllowed(projectEnv, "project.env");
|
||||
assertLowTrustEnvConfigAllowed(routineEnv, "routine.env");
|
||||
}
|
||||
const requiredScopedEnvBinding = input.requiredScopedEnvBinding ?? null;
|
||||
const requiredScopedBindingsConfigured = requiredScopedEnvBinding
|
||||
? requiredScopedEnvBinding.keys.some((key) => (
|
||||
requiredScopedEnvBinding.consumerScopes.includes("agent")
|
||||
&& isConfiguredEnvBindingValue(agentEnv[key])
|
||||
) || (
|
||||
requiredScopedEnvBinding.consumerScopes.includes("project")
|
||||
&& isConfiguredEnvBindingValue(projectEnv?.[key])
|
||||
))
|
||||
: false;
|
||||
if (requiredScopedEnvBinding && !requiredScopedBindingsConfigured) {
|
||||
throw new ConfigurationIncompleteFailure(`configuration incomplete: ${requiredScopedEnvBinding.remediation}`, {
|
||||
configurationIncomplete: {
|
||||
reason: requiredScopedEnvBinding.reason,
|
||||
companyId: input.companyId,
|
||||
agentId: input.agentId ?? null,
|
||||
issueId: input.issueId ?? null,
|
||||
projectId: input.projectId ?? null,
|
||||
routineId: input.routineId ?? null,
|
||||
requiredEnvKeys: requiredScopedEnvBinding.keys,
|
||||
requiredScopes: requiredScopedEnvBinding.consumerScopes,
|
||||
missingBindings: [],
|
||||
},
|
||||
});
|
||||
}
|
||||
// Pre-dispatch binding-validation gate: detect declared secret refs that have
|
||||
// no binding before resolving any secret value. Missing bindings short-circuit
|
||||
// to a configuration-incomplete blocker routed to a human owner instead of a
|
||||
@@ -522,6 +585,33 @@ export async function resolveExecutionRunAdapterConfig(input: {
|
||||
)),
|
||||
);
|
||||
}
|
||||
if (requiredScopedEnvBinding) {
|
||||
const requiredEnvKeys = new Set(requiredScopedEnvBinding.keys);
|
||||
const requiredScopes = new Set(requiredScopedEnvBinding.consumerScopes);
|
||||
const requiredMissingBindings = missingBindings.filter((binding) =>
|
||||
requiredScopes.has(binding.consumerType as "agent" | "project")
|
||||
&& requiredEnvKeys.has(binding.envKey),
|
||||
);
|
||||
if (requiredMissingBindings.length > 0) {
|
||||
const detail = requiredMissingBindings.map(formatMissingBindingForOperator).join("; ");
|
||||
throw new ConfigurationIncompleteFailure(
|
||||
`configuration incomplete: ${requiredScopedEnvBinding.remediation}; ${detail}`,
|
||||
{
|
||||
configurationIncomplete: {
|
||||
reason: requiredScopedEnvBinding.reason,
|
||||
companyId: input.companyId,
|
||||
agentId: input.agentId ?? null,
|
||||
issueId: input.issueId ?? null,
|
||||
projectId: input.projectId ?? null,
|
||||
routineId: input.routineId ?? null,
|
||||
requiredEnvKeys: requiredScopedEnvBinding.keys,
|
||||
requiredScopes: requiredScopedEnvBinding.consumerScopes,
|
||||
missingBindings: requiredMissingBindings,
|
||||
},
|
||||
},
|
||||
);
|
||||
}
|
||||
}
|
||||
if (missingBindings.length > 0) {
|
||||
const detail = missingBindings.map(formatMissingBindingForOperator).join("; ");
|
||||
throw new ConfigurationIncompleteFailure(`configuration incomplete: ${detail}`, {
|
||||
@@ -1080,6 +1170,53 @@ function sameResolvedPath(left: string | null | undefined, right: string | null
|
||||
return path.resolve(leftPath) === path.resolve(rightPath);
|
||||
}
|
||||
|
||||
async function hasGitPushRemote(cwd: string | null | undefined) {
|
||||
const normalized = readNonEmptyString(cwd);
|
||||
if (!normalized) return false;
|
||||
const remoteNames = await execFile("git", ["remote"], { cwd: normalized })
|
||||
.then((result) =>
|
||||
result.stdout
|
||||
.split(/\r?\n/)
|
||||
.map((value) => value.trim())
|
||||
.filter((value) => value.length > 0),
|
||||
)
|
||||
.catch(() => []);
|
||||
|
||||
for (const remoteName of remoteNames) {
|
||||
const pushUrl = await execFile("git", ["remote", "get-url", "--push", remoteName], { cwd: normalized })
|
||||
.then((result) => readNonEmptyString(result.stdout))
|
||||
.catch(() => null);
|
||||
if (pushUrl) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
export async function assertPushCapabilityCheckoutValid(input: {
|
||||
enabled: boolean;
|
||||
issue: {
|
||||
id: string;
|
||||
identifier: string | null;
|
||||
} | null;
|
||||
cwd: string | null | undefined;
|
||||
}) {
|
||||
if (!input.enabled || !input.issue) return;
|
||||
const cwd = readNonEmptyString(input.cwd);
|
||||
if (!cwd) return;
|
||||
if (await hasGitPushRemote(cwd)) return;
|
||||
throw new WorkspaceValidationFailure(
|
||||
`Issue ${input.issue.identifier ?? input.issue.id} requested the GitHub PR workflow, but checkout "${cwd}" has no configured push remote. Bind the run to a writable repo checkout before dispatching the agent.`,
|
||||
{
|
||||
workspaceValidation: {
|
||||
reason: "missing_git_push_remote",
|
||||
issueId: input.issue.id,
|
||||
issueIdentifier: input.issue.identifier,
|
||||
executionWorkspaceCwd: cwd,
|
||||
requiredEnvKeys: [...PUSH_CAPABILITY_ENV_KEYS],
|
||||
},
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
export async function assertGitSensitiveAdapterWorkspaceValid(input: {
|
||||
adapterType: string;
|
||||
agentId: string;
|
||||
@@ -8631,6 +8768,16 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
: selectedEnvironmentId
|
||||
? await environmentsSvc.getById(selectedEnvironmentId)
|
||||
: null;
|
||||
const runScopedMentionedSkillKeys = await resolveRunScopedMentionedSkillKeys({
|
||||
db,
|
||||
companyId: agent.companyId,
|
||||
issueId,
|
||||
});
|
||||
const pushCapabilityPreflightRequired = requiresPushCapabilityPreflight({
|
||||
adapterType: agent.adapterType,
|
||||
issueId,
|
||||
explicitRunScopedSkillKeys: runScopedMentionedSkillKeys,
|
||||
});
|
||||
const { resolvedConfig, secretKeys, secretManifest } = await resolveExecutionRunAdapterConfig({
|
||||
companyId: agent.companyId,
|
||||
agentId: agent.id,
|
||||
@@ -8645,6 +8792,15 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
routineEnv: routineEnvContext.env,
|
||||
secretsSvc,
|
||||
trustPreset,
|
||||
requiredScopedEnvBinding: pushCapabilityPreflightRequired
|
||||
? {
|
||||
keys: [...PUSH_CAPABILITY_ENV_KEYS],
|
||||
consumerScopes: ["agent", "project"],
|
||||
reason: "push_write_credential_missing",
|
||||
remediation:
|
||||
"GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN bound at project or agent scope.",
|
||||
}
|
||||
: undefined,
|
||||
});
|
||||
if (secretManifest.length > 0) {
|
||||
context.paperclipSecrets = {
|
||||
@@ -8653,11 +8809,6 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
} else {
|
||||
delete context.paperclipSecrets;
|
||||
}
|
||||
const runScopedMentionedSkillKeys = await resolveRunScopedMentionedSkillKeys({
|
||||
db,
|
||||
companyId: agent.companyId,
|
||||
issueId,
|
||||
});
|
||||
const effectiveResolvedConfig = applyRunScopedMentionedSkillKeys(
|
||||
resolvedConfig,
|
||||
runScopedMentionedSkillKeys,
|
||||
@@ -9271,6 +9422,16 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
environmentDriver: selectedEnvironment.driver,
|
||||
leaseMetadata: activeEnvironmentLease.lease.metadata,
|
||||
});
|
||||
await assertPushCapabilityCheckoutValid({
|
||||
enabled: pushCapabilityPreflightRequired && executionTarget?.kind === "local",
|
||||
issue: issueRef
|
||||
? {
|
||||
id: issueRef.id,
|
||||
identifier: issueRef.identifier,
|
||||
}
|
||||
: null,
|
||||
cwd: executionWorkspace.cwd,
|
||||
});
|
||||
const adapterEnv = Object.fromEntries(
|
||||
Object.entries(parseObject(resolvedConfig.env)).filter(
|
||||
(entry): entry is [string, string] => typeof entry[0] === "string" && typeof entry[1] === "string",
|
||||
|
||||
Reference in New Issue
Block a user