Fix sandbox git publishing and large workspace uploads (#8422)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox and SSH runtimes need to preserve agent work across isolated execution environments > - Git-backed workspaces were being copied mostly as filesystem archives, which breaks when `.git` points outside the mounted workspace and makes sandbox agents unable to publish their own branches > - Large ignored dependency trees could also be swept into the sandbox overlay, causing multi-GB transfers and max-string failures in some sandbox clients > - This pull request makes sandbox runtime setup use a git-backed HEAD sync plus a small dirty/untracked overlay, and bounds sandbox file transfers so large archives do not need one huge string > - The benefit is that sandbox agents can commit and push from a usable git checkout without uploading dependency trees such as `node_modules` ## Linked Issues or Issue Description Refs #8395 No public duplicate issue or PR was found after searches for `sandbox git workspace`, `git push sandbox`, and `node_modules sandbox upload`. Bug report: - What happened: sandbox-backed agent workspaces could receive a `.git` file that pointed at host-only git state, leaving the sandbox unable to run normal git workflows. The sandbox overlay upload could also include ignored dependency directories, creating very large transfers. - Expected behavior: sandbox and remote runtimes should prepare a usable git-backed workspace, copy only the necessary workspace overlay, and restore git history plus file changes without depending on a host-only `.git` path. - Steps to reproduce: 1. Run an agent in a sandbox-backed workspace whose local git checkout is a worktree. 2. Ask the agent to complete a GitHub workflow that requires commit/push access. 3. Observe that git operations can fail inside the sandbox, and ignored dependency trees can be uploaded as part of the workspace overlay. - Paperclip version or commit: reproduced against `master` before this PR, base `7aa212296eb1`. - Deployment mode: local dev / sandbox-backed runtime. - Installation method: built from source. - Agent adapters involved: local adapters using shared adapter-utils runtime preparation. - Database mode: not database-related. - Access context: agent runtime. - Local verification environment: Node.js v25.6.1, pnpm 9.15.4, macOS arm64. - Privacy checklist: all pasted output was reviewed for secrets, private hostnames, local usernames, and internal instance links. ## What Changed - Added a GitHub workflow push preflight so agent runs can detect missing push credentials when a workflow explicitly needs GitHub publishing. - Added shared git workspace sync helpers for shallow HEAD import/export and dirty/untracked overlay tracking. - Updated sandbox managed runtime setup to use git history plus a selected overlay instead of uploading the full local workspace for git-backed workspaces. - Bounded sandbox archive upload/download paths so large payloads stream or chunk instead of materializing one oversized string. - Excluded `.git` and ignored dependency trees from sandbox upload, download, and restore baselines while preserving local ignored directories during sync-back. - Added focused tests for git workspace sync, sandbox overlay selection, transfer chunking, and heartbeat push-preflight behavior. ## Verification - `pnpm exec vitest run packages/adapter-utils/src/git-workspace-sync.test.ts packages/adapter-utils/src/sandbox-managed-runtime.test.ts packages/adapter-utils/src/command-managed-runtime.test.ts server/src/__tests__/heartbeat-project-env.test.ts server/src/__tests__/heartbeat-workspace-session.test.ts` - `pnpm --filter @paperclipai/adapter-utils typecheck` - `pnpm --filter @paperclipai/server typecheck` - `pnpm build` - Public-hygiene scan of the PR diff and commit messages for internal issue ids, local paths, private hostnames, and obvious token patterns. ## Risks - Medium risk: this changes sandbox runtime synchronization semantics for git-backed workspaces, especially around dirty tracked files, untracked files, deleted paths, and ignored files. - The main mitigation is focused test coverage for upload contents, restore exclusions, and git round-trip behavior. - The SSH runtime keeps the current bundle-based implementation from `master`; this PR only aligns shared excludes and sandbox behavior with that model. ## Model Used OpenAI Codex, GPT-5-based coding agent, tool-enabled shell/git/GitHub workflow, with code execution and repository inspection. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
@@ -22,7 +22,10 @@ interface SpawnRunnerHandle {
|
|||||||
// A runner that actually executes the shell scripts (piping stdin through a real
|
// A runner that actually executes the shell scripts (piping stdin through a real
|
||||||
// pipe so multi-MB payloads work) and replays stdout through onLog in several
|
// pipe so multi-MB payloads work) and replays stdout through onLog in several
|
||||||
// chunks so the streaming readFile byte-counter is exercised.
|
// chunks so the streaming readFile byte-counter is exercised.
|
||||||
function makeSpawnRunner(options: { supportsSingleStreamStdinProgress?: boolean } = {}): SpawnRunnerHandle {
|
function makeSpawnRunner(options: {
|
||||||
|
supportsSingleStreamStdinProgress?: boolean;
|
||||||
|
maxStdoutBytes?: number;
|
||||||
|
} = {}): SpawnRunnerHandle {
|
||||||
const calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }> = [];
|
const calls: Array<{ command: string; args?: string[]; cwd?: string; stdin?: string }> = [];
|
||||||
const runner: CommandManagedRuntimeRunner = {
|
const runner: CommandManagedRuntimeRunner = {
|
||||||
supportsSingleStreamStdinProgress: options.supportsSingleStreamStdinProgress,
|
supportsSingleStreamStdinProgress: options.supportsSingleStreamStdinProgress,
|
||||||
@@ -48,6 +51,21 @@ function makeSpawnRunner(options: { supportsSingleStreamStdinProgress?: boolean
|
|||||||
resolve({ exitCode: 127, signal: null, timedOut: false, stdout, stderr, pid: null, startedAt });
|
resolve({ exitCode: 127, signal: null, timedOut: false, stdout, stderr, pid: null, startedAt });
|
||||||
});
|
});
|
||||||
child.on("close", async (code) => {
|
child.on("close", async (code) => {
|
||||||
|
if (
|
||||||
|
options.maxStdoutBytes != null &&
|
||||||
|
Buffer.byteLength(stdout, "utf8") > options.maxStdoutBytes
|
||||||
|
) {
|
||||||
|
resolve({
|
||||||
|
exitCode: 1,
|
||||||
|
signal: null,
|
||||||
|
timedOut: false,
|
||||||
|
stdout,
|
||||||
|
stderr: `stdout exceeded ${options.maxStdoutBytes} bytes`,
|
||||||
|
pid: child.pid ?? null,
|
||||||
|
startedAt,
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
}
|
||||||
if (input.onLog && stdout.length > 0) {
|
if (input.onLog && stdout.length > 0) {
|
||||||
const chunkSize = Math.max(1, Math.ceil(stdout.length / 4));
|
const chunkSize = Math.max(1, Math.ceil(stdout.length / 4));
|
||||||
for (let offset = 0; offset < stdout.length; offset += chunkSize) {
|
for (let offset = 0; offset < stdout.length; offset += chunkSize) {
|
||||||
@@ -75,6 +93,26 @@ function toArrayBuffer(buffer: Buffer): ArrayBuffer {
|
|||||||
return Uint8Array.from(buffer).buffer;
|
return Uint8Array.from(buffer).buffer;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function withBase64StringByteLimit<T>(limitBytes: number, fn: () => Promise<T>): Promise<T> {
|
||||||
|
const originalToString = Buffer.prototype.toString;
|
||||||
|
Buffer.prototype.toString = function patchedToString(
|
||||||
|
this: Buffer,
|
||||||
|
encoding?: BufferEncoding,
|
||||||
|
start?: number,
|
||||||
|
end?: number,
|
||||||
|
) {
|
||||||
|
if (encoding === "base64" && this.byteLength > limitBytes) {
|
||||||
|
throw new Error(`test guard: attempted to base64-encode ${this.byteLength} bytes at once`);
|
||||||
|
}
|
||||||
|
return originalToString.call(this, encoding, start, end);
|
||||||
|
} as typeof Buffer.prototype.toString;
|
||||||
|
try {
|
||||||
|
return await fn();
|
||||||
|
} finally {
|
||||||
|
Buffer.prototype.toString = originalToString;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
describe("command managed runtime", () => {
|
describe("command managed runtime", () => {
|
||||||
const cleanupDirs: string[] = [];
|
const cleanupDirs: string[] = [];
|
||||||
|
|
||||||
@@ -243,10 +281,12 @@ describe("command managed runtime", () => {
|
|||||||
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
||||||
|
|
||||||
const progress: Array<{ done: number; total: number | null }> = [];
|
const progress: Array<{ done: number; total: number | null }> = [];
|
||||||
await client.writeFile(remotePath, toArrayBuffer(payload), {
|
await withBase64StringByteLimit(4 * 1024 * 1024, async () => {
|
||||||
onProgress: (done, total) => {
|
await client.writeFile(remotePath, toArrayBuffer(payload), {
|
||||||
progress.push({ done, total });
|
onProgress: (done, total) => {
|
||||||
},
|
progress.push({ done, total });
|
||||||
|
},
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// Exactly one upload process: O(1) round-trips regardless of payload size.
|
// Exactly one upload process: O(1) round-trips regardless of payload size.
|
||||||
@@ -288,6 +328,9 @@ describe("command managed runtime", () => {
|
|||||||
// Provider-backed sandbox runners cannot surface mid-flight progress for a
|
// Provider-backed sandbox runners cannot surface mid-flight progress for a
|
||||||
// single stdin RPC, so we intentionally use several large append commands.
|
// single stdin RPC, so we intentionally use several large append commands.
|
||||||
expect(calls.length).toBeGreaterThan(2);
|
expect(calls.length).toBeGreaterThan(2);
|
||||||
|
const stdinCalls = calls.filter((call) => call.stdin != null);
|
||||||
|
expect(stdinCalls.length).toBeGreaterThan(2);
|
||||||
|
expect(stdinCalls.every((call) => Buffer.byteLength(call.stdin ?? "", "utf8") <= 4.1 * 1024 * 1024)).toBe(true);
|
||||||
expect(progress.length).toBeGreaterThan(2);
|
expect(progress.length).toBeGreaterThan(2);
|
||||||
for (let i = 1; i < progress.length; i++) {
|
for (let i = 1; i < progress.length; i++) {
|
||||||
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
|
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
|
||||||
@@ -295,16 +338,41 @@ describe("command managed runtime", () => {
|
|||||||
expect(progress.at(-1)).toEqual({ done: payload.length, total: payload.length });
|
expect(progress.at(-1)).toEqual({ done: payload.length, total: payload.length });
|
||||||
});
|
});
|
||||||
|
|
||||||
it("streams a download through onLog and reports monotonic byte progress to the total", async () => {
|
it("falls back to bounded chunks when the runner does not explicitly opt in", async () => {
|
||||||
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-write-fallback-no-progress-"));
|
||||||
|
cleanupDirs.push(rootDir);
|
||||||
|
const remotePath = path.join(rootDir, "nested", "payload.bin");
|
||||||
|
|
||||||
|
const payload = Buffer.alloc(12 * 1024 * 1024);
|
||||||
|
for (let i = 0; i < payload.length; i++) payload[i] = i % 256;
|
||||||
|
|
||||||
|
const { runner, calls } = makeSpawnRunner();
|
||||||
|
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
||||||
|
|
||||||
|
await withBase64StringByteLimit(4 * 1024 * 1024, async () => {
|
||||||
|
await client.writeFile(remotePath, toArrayBuffer(payload));
|
||||||
|
});
|
||||||
|
|
||||||
|
const written = await readFile(remotePath);
|
||||||
|
expect(written.equals(payload)).toBe(true);
|
||||||
|
|
||||||
|
// A runner that doesn't mark single-stream stdin support must avoid passing
|
||||||
|
// the whole base64 archive as one string, so we expect multiple append calls.
|
||||||
|
const stdinCalls = calls.filter((call) => call.stdin != null);
|
||||||
|
expect(stdinCalls.length).toBeGreaterThan(1);
|
||||||
|
expect(stdinCalls.every((call) => Buffer.byteLength(call.stdin ?? "", "utf8") <= 4.1 * 1024 * 1024)).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("downloads in bounded stdout chunks and reports monotonic byte progress to the total", async () => {
|
||||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-read-"));
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-command-read-"));
|
||||||
cleanupDirs.push(rootDir);
|
cleanupDirs.push(rootDir);
|
||||||
const remotePath = path.join(rootDir, "download.bin");
|
const remotePath = path.join(rootDir, "download.bin");
|
||||||
|
|
||||||
const payload = Buffer.alloc(512 * 1024);
|
const payload = Buffer.alloc(7 * 1024 * 1024);
|
||||||
for (let i = 0; i < payload.length; i++) payload[i] = (i * 7) % 256;
|
for (let i = 0; i < payload.length; i++) payload[i] = (i * 7) % 256;
|
||||||
await writeFile(remotePath, payload);
|
await writeFile(remotePath, payload);
|
||||||
|
|
||||||
const { runner } = makeSpawnRunner();
|
const { runner, calls } = makeSpawnRunner({ maxStdoutBytes: 5 * 1024 * 1024 });
|
||||||
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
const client = createCommandManagedRuntimeClient({ runner, commandCwd: "/", timeoutMs: 30_000 });
|
||||||
|
|
||||||
const progress: Array<{ done: number; total: number | null }> = [];
|
const progress: Array<{ done: number; total: number | null }> = [];
|
||||||
@@ -316,7 +384,10 @@ describe("command managed runtime", () => {
|
|||||||
|
|
||||||
expect(Buffer.from(bytes as ArrayBuffer).equals(payload)).toBe(true);
|
expect(Buffer.from(bytes as ArrayBuffer).equals(payload)).toBe(true);
|
||||||
|
|
||||||
// The runner replays stdout in several chunks, so progress arrives in steps.
|
// The old single `base64 < file` path would exceed the runner's stdout cap.
|
||||||
|
// The bounded path reads with several small `dd | base64` commands instead.
|
||||||
|
expect(calls.some((call) => call.args?.join(" ").includes("base64 <"))).toBe(false);
|
||||||
|
expect(calls.filter((call) => call.args?.join(" ").includes("dd if=")).length).toBeGreaterThan(1);
|
||||||
expect(progress.length).toBeGreaterThan(1);
|
expect(progress.length).toBeGreaterThan(1);
|
||||||
for (let i = 1; i < progress.length; i++) {
|
for (let i = 1; i < progress.length; i++) {
|
||||||
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
|
expect(progress[i].done).toBeGreaterThanOrEqual(progress[i - 1].done);
|
||||||
|
|||||||
@@ -56,6 +56,12 @@ const REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES = 96 * 1024 * 1024;
|
|||||||
// Fallback chunk size (base64 bytes). Kept a multiple of 4 so each chunk is a
|
// Fallback chunk size (base64 bytes). Kept a multiple of 4 so each chunk is a
|
||||||
// self-contained base64 unit that decodes cleanly on its own.
|
// self-contained base64 unit that decodes cleanly on its own.
|
||||||
const REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE = 4 * 1024 * 1024;
|
const REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE = 4 * 1024 * 1024;
|
||||||
|
const REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE = (REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE / 4) * 3;
|
||||||
|
const REMOTE_READ_CHUNK_BYTES = REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE;
|
||||||
|
|
||||||
|
function base64EncodedLength(byteLength: number): number {
|
||||||
|
return Math.ceil(byteLength / 3) * 4;
|
||||||
|
}
|
||||||
|
|
||||||
function toBuffer(bytes: Buffer | Uint8Array | ArrayBuffer): Buffer {
|
function toBuffer(bytes: Buffer | Uint8Array | ArrayBuffer): Buffer {
|
||||||
if (Buffer.isBuffer(bytes)) return bytes;
|
if (Buffer.isBuffer(bytes)) return bytes;
|
||||||
@@ -104,11 +110,10 @@ export function createCommandManagedRuntimeClient(input: {
|
|||||||
writeFile: async (remotePath, bytes, options) => {
|
writeFile: async (remotePath, bytes, options) => {
|
||||||
const buffer = toBuffer(bytes);
|
const buffer = toBuffer(bytes);
|
||||||
const total = buffer.byteLength;
|
const total = buffer.byteLength;
|
||||||
const body = buffer.toString("base64");
|
const encodedLength = base64EncodedLength(total);
|
||||||
const remoteDir = path.posix.dirname(remotePath);
|
const remoteDir = path.posix.dirname(remotePath);
|
||||||
const remoteTempPath = `${remotePath}.paperclip-upload`;
|
const remoteTempPath = `${remotePath}.paperclip-upload`;
|
||||||
const canUseSingleStreamProgressPath =
|
const canUseSingleStreamProgressPath = input.runner.supportsSingleStreamStdinProgress === true;
|
||||||
!options?.onProgress || input.runner.supportsSingleStreamStdinProgress === true;
|
|
||||||
|
|
||||||
// Primary path: a single round-trip. Stream the entire base64 body to one
|
// Primary path: a single round-trip. Stream the entire base64 body to one
|
||||||
// `base64 -d` process via stdin, decode straight into a temp file, then
|
// `base64 -d` process via stdin, decode straight into a temp file, then
|
||||||
@@ -116,9 +121,10 @@ export function createCommandManagedRuntimeClient(input: {
|
|||||||
// one `printf >> tmpfile` shell round-trip per 32 KB — thousands of serial
|
// one `printf >> tmpfile` shell round-trip per 32 KB — thousands of serial
|
||||||
// processes for a large workspace — with exactly one process.
|
// processes for a large workspace — with exactly one process.
|
||||||
if (
|
if (
|
||||||
body.length <= REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES &&
|
encodedLength <= REMOTE_WRITE_SINGLE_STREAM_MAX_BASE64_BYTES &&
|
||||||
canUseSingleStreamProgressPath
|
canUseSingleStreamProgressPath
|
||||||
) {
|
) {
|
||||||
|
const body = buffer.toString("base64");
|
||||||
await options?.onProgress?.(0, total);
|
await options?.onProgress?.(0, total);
|
||||||
await runShell(
|
await runShell(
|
||||||
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
||||||
@@ -139,62 +145,50 @@ export function createCommandManagedRuntimeClient(input: {
|
|||||||
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
`mkdir -p ${shellQuote(remoteDir)} && ` +
|
||||||
`rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`,
|
`rm -f ${shellQuote(remoteTempPath)} && : > ${shellQuote(remoteTempPath)}`,
|
||||||
);
|
);
|
||||||
for (let offset = 0; offset < body.length; offset += REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE) {
|
for (let offset = 0; offset < total; offset += REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE) {
|
||||||
const chunk = body.slice(offset, offset + REMOTE_WRITE_FALLBACK_BASE64_CHUNK_SIZE);
|
const end = Math.min(total, offset + REMOTE_WRITE_FALLBACK_DECODED_CHUNK_SIZE);
|
||||||
|
const chunk = buffer.subarray(offset, end).toString("base64");
|
||||||
await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk });
|
await runShell(`base64 -d >> ${shellQuote(remoteTempPath)}`, { stdin: chunk });
|
||||||
const decodedSoFar = Math.min(total, Math.floor(((offset + chunk.length) * 3) / 4));
|
await options?.onProgress?.(end, total);
|
||||||
await options?.onProgress?.(decodedSoFar, total);
|
|
||||||
}
|
}
|
||||||
await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`);
|
await runShell(`mv -f ${shellQuote(remoteTempPath)} ${shellQuote(remotePath)}`);
|
||||||
await options?.onProgress?.(total, total);
|
await options?.onProgress?.(total, total);
|
||||||
},
|
},
|
||||||
readFile: async (remotePath, options) => {
|
readFile: async (remotePath, options) => {
|
||||||
// Decoded file size up front so download progress can be a percentage.
|
// Chunked reads intentionally query the remote size first, even without
|
||||||
// Only paid when a progress hook is attached.
|
// a progress sink, so each sandbox RPC stays bounded and truncation is
|
||||||
let totalBytes: number | null = null;
|
// detected without materializing the whole file as one stdout string.
|
||||||
if (options?.onProgress) {
|
const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`);
|
||||||
const sizeResult = await runShell(`wc -c < ${shellQuote(remotePath)}`);
|
const totalBytes = Number.parseInt(sizeResult.stdout.trim(), 10);
|
||||||
const parsed = Number.parseInt(sizeResult.stdout.trim(), 10);
|
if (!Number.isFinite(totalBytes) || totalBytes < 0) {
|
||||||
totalBytes = Number.isFinite(parsed) && parsed >= 0 ? parsed : null;
|
throw new Error(`Could not determine remote file size for ${remotePath}`);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stream the remote `base64` stdout through a byte counter, decoding each
|
// Read in bounded remote chunks so the runner never has to materialize a
|
||||||
// 4-char-aligned slice incrementally rather than buffering the whole
|
// single base64 stdout string for the whole archive. The client API still
|
||||||
// response as one string. Falls back to the buffered result when the
|
// returns the decoded file as a Buffer, but every command result stays
|
||||||
// runner does not surface incremental stdout.
|
// small enough for provider-backed sandbox RPCs.
|
||||||
const decodedChunks: Buffer[] = [];
|
const decodedChunks: Buffer[] = [];
|
||||||
let b64Remainder = "";
|
|
||||||
let decodedSoFar = 0;
|
let decodedSoFar = 0;
|
||||||
let streamed = false;
|
if (totalBytes === 0) {
|
||||||
const result = await runShell(`base64 < ${shellQuote(remotePath)}`, {
|
await options?.onProgress?.(0, 0);
|
||||||
onLog: async (stream, chunk) => {
|
return Buffer.alloc(0);
|
||||||
if (stream !== "stdout") return;
|
|
||||||
streamed = true;
|
|
||||||
const data = b64Remainder + chunk.replace(/\s+/g, "");
|
|
||||||
const alignedLen = data.length - (data.length % 4);
|
|
||||||
if (alignedLen > 0) {
|
|
||||||
const decoded = Buffer.from(data.slice(0, alignedLen), "base64");
|
|
||||||
decodedChunks.push(decoded);
|
|
||||||
decodedSoFar += decoded.byteLength;
|
|
||||||
}
|
|
||||||
b64Remainder = data.slice(alignedLen);
|
|
||||||
if (options?.onProgress) {
|
|
||||||
const done = totalBytes != null ? Math.min(totalBytes, decodedSoFar) : decodedSoFar;
|
|
||||||
await options.onProgress(done, totalBytes);
|
|
||||||
}
|
|
||||||
},
|
|
||||||
});
|
|
||||||
|
|
||||||
let out: Buffer;
|
|
||||||
if (streamed) {
|
|
||||||
if (b64Remainder.length > 0) {
|
|
||||||
decodedChunks.push(Buffer.from(b64Remainder, "base64"));
|
|
||||||
}
|
|
||||||
out = Buffer.concat(decodedChunks);
|
|
||||||
} else {
|
|
||||||
out = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
|
|
||||||
}
|
}
|
||||||
await options?.onProgress?.(out.byteLength, totalBytes ?? out.byteLength);
|
for (let chunkIndex = 0; decodedSoFar < totalBytes; chunkIndex++) {
|
||||||
|
const result = await runShell(
|
||||||
|
`dd if=${shellQuote(remotePath)} bs=${REMOTE_READ_CHUNK_BYTES} skip=${chunkIndex} count=1 2>/dev/null | base64`,
|
||||||
|
);
|
||||||
|
const chunk = Buffer.from(result.stdout.replace(/\s+/g, ""), "base64");
|
||||||
|
if (chunk.byteLength === 0) break;
|
||||||
|
decodedChunks.push(chunk);
|
||||||
|
decodedSoFar += chunk.byteLength;
|
||||||
|
await options?.onProgress?.(Math.min(decodedSoFar, totalBytes), totalBytes);
|
||||||
|
}
|
||||||
|
const out = Buffer.concat(decodedChunks);
|
||||||
|
if (out.byteLength !== totalBytes) {
|
||||||
|
throw new Error(`Remote file read was truncated for ${remotePath}: ${out.byteLength}/${totalBytes} bytes`);
|
||||||
|
}
|
||||||
|
await options?.onProgress?.(out.byteLength, totalBytes);
|
||||||
return out;
|
return out;
|
||||||
},
|
},
|
||||||
listFiles: async (remotePath) => {
|
listFiles: async (remotePath) => {
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
export function isRelativePathOrDescendant(relative: string, candidate: string): boolean {
|
||||||
|
return relative === candidate || relative.startsWith(`${candidate}/`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function pathContainsSegmentOrDescendant(relative: string, segment: string): boolean {
|
||||||
|
return relative === segment ||
|
||||||
|
relative.startsWith(`${segment}/`) ||
|
||||||
|
relative.endsWith(`/${segment}`) ||
|
||||||
|
relative.includes(`/${segment}/`);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function excludePatternMatches(relative: string, pattern: string): boolean {
|
||||||
|
if (pattern.startsWith("*/") && pattern.endsWith("/*")) {
|
||||||
|
return pathContainsSegmentOrDescendant(relative, pattern.slice(2, -2));
|
||||||
|
}
|
||||||
|
if (pattern.startsWith("*/")) {
|
||||||
|
return pathContainsSegmentOrDescendant(relative, pattern.slice(2));
|
||||||
|
}
|
||||||
|
if (pattern.endsWith("/*")) {
|
||||||
|
const base = pattern.slice(0, -2);
|
||||||
|
return relative.startsWith(`${base}/`);
|
||||||
|
}
|
||||||
|
return isRelativePathOrDescendant(relative, pattern);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function shouldExcludePath(relative: string, exclude: readonly string[]): boolean {
|
||||||
|
return exclude.some((entry) => excludePatternMatches(relative, entry));
|
||||||
|
}
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
import { execFile as execFileCallback } from "node:child_process";
|
||||||
|
import { lstat, mkdir, mkdtemp, readFile, rm, stat, writeFile } from "node:fs/promises";
|
||||||
|
import os from "node:os";
|
||||||
|
import path from "node:path";
|
||||||
|
import { promisify } from "node:util";
|
||||||
|
import { afterEach, describe, expect, it } from "vitest";
|
||||||
|
|
||||||
|
import {
|
||||||
|
buildRemoteGitDeltaBundleScript,
|
||||||
|
createImportedGitRef,
|
||||||
|
createRemoteGitExportRef,
|
||||||
|
deleteLocalGitRef,
|
||||||
|
fetchGitBundleIntoLocalRef,
|
||||||
|
readGitWorkspaceSnapshot,
|
||||||
|
runLocalGit,
|
||||||
|
withShallowGitWorkspaceClone,
|
||||||
|
} from "./git-workspace-sync.js";
|
||||||
|
|
||||||
|
const execFile = promisify(execFileCallback);
|
||||||
|
|
||||||
|
async function git(cwd: string, args: string[]): Promise<string> {
|
||||||
|
return (await runLocalGit(cwd, args)).stdout.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("git workspace sync", () => {
|
||||||
|
const cleanupDirs: string[] = [];
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
while (cleanupDirs.length > 0) {
|
||||||
|
const dir = cleanupDirs.pop();
|
||||||
|
if (!dir) continue;
|
||||||
|
await rm(dir, { recursive: true, force: true }).catch(() => undefined);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
async function createRepo(rootDir: string): Promise<string> {
|
||||||
|
const repo = path.join(rootDir, "repo");
|
||||||
|
await mkdir(repo, { recursive: true });
|
||||||
|
await git(repo, ["init"]);
|
||||||
|
await git(repo, ["checkout", "-b", "main"]);
|
||||||
|
await git(repo, ["config", "user.name", "Paperclip Test"]);
|
||||||
|
await git(repo, ["config", "user.email", "test@paperclip.dev"]);
|
||||||
|
await writeFile(path.join(repo, "tracked.txt"), "base\n", "utf8");
|
||||||
|
await git(repo, ["add", "tracked.txt"]);
|
||||||
|
await git(repo, ["commit", "-m", "base"]);
|
||||||
|
return repo;
|
||||||
|
}
|
||||||
|
|
||||||
|
it("creates a shallow standalone clone from the local HEAD snapshot", async () => {
|
||||||
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-sync-"));
|
||||||
|
cleanupDirs.push(rootDir);
|
||||||
|
const repo = await createRepo(rootDir);
|
||||||
|
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
|
||||||
|
await rm(path.join(repo, "tracked.txt"));
|
||||||
|
|
||||||
|
const snapshot = await readGitWorkspaceSnapshot(repo);
|
||||||
|
expect(snapshot).toMatchObject({
|
||||||
|
headCommit: baseHead,
|
||||||
|
branchName: "main",
|
||||||
|
deletedPaths: ["tracked.txt"],
|
||||||
|
});
|
||||||
|
|
||||||
|
await withShallowGitWorkspaceClone({
|
||||||
|
localDir: repo,
|
||||||
|
snapshot: snapshot!,
|
||||||
|
}, async (cloneDir) => {
|
||||||
|
expect((await lstat(path.join(cloneDir, ".git"))).isDirectory()).toBe(true);
|
||||||
|
await expect(readFile(path.join(cloneDir, ".git", "shallow"), "utf8")).resolves.toContain(baseHead);
|
||||||
|
expect(await git(cloneDir, ["rev-list", "--count", "HEAD"])).toBe("1");
|
||||||
|
expect(await git(cloneDir, ["branch", "--show-current"])).toBe("main");
|
||||||
|
await expect(readFile(path.join(cloneDir, "tracked.txt"), "utf8")).resolves.toBe("base\n");
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("builds thin git delta bundles relative to the imported base", async () => {
|
||||||
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-delta-"));
|
||||||
|
cleanupDirs.push(rootDir);
|
||||||
|
const repo = await createRepo(rootDir);
|
||||||
|
const baseHead = await git(repo, ["rev-parse", "HEAD"]);
|
||||||
|
const snapshot = await readGitWorkspaceSnapshot(repo);
|
||||||
|
expect(snapshot).not.toBeNull();
|
||||||
|
|
||||||
|
await withShallowGitWorkspaceClone({
|
||||||
|
localDir: repo,
|
||||||
|
snapshot: snapshot!,
|
||||||
|
}, async (remoteDir) => {
|
||||||
|
const emptyBundle = path.join(rootDir, "empty.bundle");
|
||||||
|
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
|
||||||
|
remoteDir,
|
||||||
|
baseSha: baseHead,
|
||||||
|
exportRef: createRemoteGitExportRef("test"),
|
||||||
|
bundlePath: emptyBundle,
|
||||||
|
})]);
|
||||||
|
expect((await stat(emptyBundle)).size).toBe(0);
|
||||||
|
|
||||||
|
await git(remoteDir, ["config", "user.name", "Paperclip Remote"]);
|
||||||
|
await git(remoteDir, ["config", "user.email", "remote@paperclip.dev"]);
|
||||||
|
await writeFile(path.join(remoteDir, "tracked.txt"), "remote\n", "utf8");
|
||||||
|
await git(remoteDir, ["commit", "-am", "remote update"]);
|
||||||
|
const remoteHead = await git(remoteDir, ["rev-parse", "HEAD"]);
|
||||||
|
|
||||||
|
const deltaBundle = path.join(rootDir, "delta.bundle");
|
||||||
|
const importedRef = createImportedGitRef("test");
|
||||||
|
const exportRef = createRemoteGitExportRef("test");
|
||||||
|
try {
|
||||||
|
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
|
||||||
|
remoteDir,
|
||||||
|
baseSha: baseHead,
|
||||||
|
exportRef,
|
||||||
|
bundlePath: deltaBundle,
|
||||||
|
})]);
|
||||||
|
expect((await stat(deltaBundle)).size).toBeGreaterThan(0);
|
||||||
|
|
||||||
|
const importedHead = await fetchGitBundleIntoLocalRef({
|
||||||
|
localDir: repo,
|
||||||
|
bundlePath: deltaBundle,
|
||||||
|
exportRef,
|
||||||
|
importedRef,
|
||||||
|
baseSha: baseHead,
|
||||||
|
});
|
||||||
|
expect(importedHead).toBe(remoteHead);
|
||||||
|
expect(await git(repo, ["rev-list", "--count", importedRef, "--not", baseHead])).toBe("1");
|
||||||
|
} finally {
|
||||||
|
await deleteLocalGitRef({ localDir: repo, ref: importedRef });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,321 @@
|
|||||||
|
import { randomUUID } from "node:crypto";
|
||||||
|
import { execFile } from "node:child_process";
|
||||||
|
import { promises as fs } from "node:fs";
|
||||||
|
import os from "node:os";
|
||||||
|
import path from "node:path";
|
||||||
|
|
||||||
|
export interface GitCommandResult {
|
||||||
|
stdout: string;
|
||||||
|
stderr: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface GitWorkspaceSnapshot {
|
||||||
|
headCommit: string;
|
||||||
|
branchName: string | null;
|
||||||
|
overlayPaths: string[];
|
||||||
|
deletedPaths: string[];
|
||||||
|
ignoredPaths: string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export const GIT_ARCHIVE_EXCLUDES = [".git", ".git/*"] as const;
|
||||||
|
|
||||||
|
function shellQuote(value: string) {
|
||||||
|
return `'${value.replace(/'/g, `'\"'\"'`)}'`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function runLocalGit(
|
||||||
|
localDir: string,
|
||||||
|
args: string[],
|
||||||
|
options: {
|
||||||
|
timeout?: number;
|
||||||
|
maxBuffer?: number;
|
||||||
|
} = {},
|
||||||
|
): Promise<GitCommandResult> {
|
||||||
|
return await new Promise<GitCommandResult>((resolve, reject) => {
|
||||||
|
execFile(
|
||||||
|
"git",
|
||||||
|
["-C", localDir, ...args],
|
||||||
|
{
|
||||||
|
timeout: options.timeout ?? 15_000,
|
||||||
|
maxBuffer: options.maxBuffer ?? 1024 * 128,
|
||||||
|
},
|
||||||
|
(error, stdout, stderr) => {
|
||||||
|
if (error) {
|
||||||
|
reject(Object.assign(error, { stdout: stdout ?? "", stderr: stderr ?? "" }));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
resolve({
|
||||||
|
stdout: stdout ?? "",
|
||||||
|
stderr: stderr ?? "",
|
||||||
|
});
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function readGitWorkspaceSnapshot(localDir: string): Promise<GitWorkspaceSnapshot | null> {
|
||||||
|
try {
|
||||||
|
const insideWorkTree = await runLocalGit(localDir, ["rev-parse", "--is-inside-work-tree"], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
});
|
||||||
|
if (insideWorkTree.stdout.trim() !== "true") {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
const [headCommitResult, branchResult, overlayDiffResult, untrackedResult, deletedResult, ignoredResult] = await Promise.all([
|
||||||
|
runLocalGit(localDir, ["rev-parse", "HEAD"], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
}),
|
||||||
|
runLocalGit(localDir, ["rev-parse", "--abbrev-ref", "HEAD"], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
}),
|
||||||
|
runLocalGit(localDir, ["diff", "--name-only", "-z", "--diff-filter=ACMRTUXB", "HEAD", "--"], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 1024 * 1024,
|
||||||
|
}),
|
||||||
|
runLocalGit(localDir, ["ls-files", "--others", "--exclude-standard", "-z"], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 1024 * 1024,
|
||||||
|
}),
|
||||||
|
runLocalGit(localDir, ["diff", "--name-only", "-z", "--diff-filter=D", "HEAD", "--"], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 256 * 1024,
|
||||||
|
}),
|
||||||
|
runLocalGit(localDir, ["status", "--ignored", "--porcelain=v1", "-z", "--untracked-files=normal"], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 1024 * 1024,
|
||||||
|
}),
|
||||||
|
]);
|
||||||
|
|
||||||
|
const branchName = branchResult.stdout.trim();
|
||||||
|
const splitNul = (value: string) => value.split("\0").map((entry) => entry.trim()).filter(Boolean);
|
||||||
|
return {
|
||||||
|
headCommit: headCommitResult.stdout.trim(),
|
||||||
|
branchName: branchName && branchName !== "HEAD" ? branchName : null,
|
||||||
|
overlayPaths: [...new Set([...splitNul(overlayDiffResult.stdout), ...splitNul(untrackedResult.stdout)])]
|
||||||
|
.sort((left, right) => left.localeCompare(right)),
|
||||||
|
deletedPaths: [...new Set(splitNul(deletedResult.stdout))]
|
||||||
|
.sort((left, right) => left.localeCompare(right)),
|
||||||
|
ignoredPaths: splitNul(ignoredResult.stdout)
|
||||||
|
.filter((entry) => entry.startsWith("!! "))
|
||||||
|
.map((entry) => entry.slice(3).replace(/\/+$/, ""))
|
||||||
|
.filter(Boolean)
|
||||||
|
.sort((left, right) => left.localeCompare(right)),
|
||||||
|
};
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function withShallowGitWorkspaceClone<T>(
|
||||||
|
input: {
|
||||||
|
localDir: string;
|
||||||
|
snapshot: GitWorkspaceSnapshot;
|
||||||
|
},
|
||||||
|
fn: (cloneDir: string) => Promise<T>,
|
||||||
|
): Promise<T> {
|
||||||
|
const cloneDir = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-workspace-"));
|
||||||
|
const tempRef = `refs/paperclip/git-sync/import/${randomUUID()}`;
|
||||||
|
try {
|
||||||
|
await runLocalGit(input.localDir, ["update-ref", tempRef, input.snapshot.headCommit], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
});
|
||||||
|
await runLocalGit(cloneDir, ["init"], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 64 * 1024,
|
||||||
|
});
|
||||||
|
await runLocalGit(cloneDir, ["fetch", "--depth=1", input.localDir, tempRef], {
|
||||||
|
timeout: 60_000,
|
||||||
|
maxBuffer: 1024 * 1024,
|
||||||
|
});
|
||||||
|
await runLocalGit(
|
||||||
|
cloneDir,
|
||||||
|
input.snapshot.branchName
|
||||||
|
? ["checkout", "--force", "-B", input.snapshot.branchName, "FETCH_HEAD"]
|
||||||
|
: ["checkout", "--force", "--detach", "FETCH_HEAD"],
|
||||||
|
{
|
||||||
|
timeout: 60_000,
|
||||||
|
maxBuffer: 1024 * 1024,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
await runLocalGit(cloneDir, ["reset", "--hard", input.snapshot.headCommit], {
|
||||||
|
timeout: 60_000,
|
||||||
|
maxBuffer: 1024 * 1024,
|
||||||
|
});
|
||||||
|
return await fn(cloneDir);
|
||||||
|
} finally {
|
||||||
|
await runLocalGit(input.localDir, ["update-ref", "-d", tempRef], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
}).catch(() => undefined);
|
||||||
|
await fs.rm(cloneDir, { recursive: true, force: true }).catch(() => undefined);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createImportedGitRef(scope = "remote"): string {
|
||||||
|
return `refs/paperclip/git-sync/imported/${scope}/${randomUUID()}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function createRemoteGitExportRef(scope = "remote"): string {
|
||||||
|
return `refs/paperclip/git-sync/export/${scope}/${randomUUID()}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function deleteLocalGitRef(input: {
|
||||||
|
localDir: string;
|
||||||
|
ref: string;
|
||||||
|
}): Promise<void> {
|
||||||
|
await runLocalGit(input.localDir, ["update-ref", "-d", input.ref], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
}).catch(() => undefined);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function fetchGitBundleIntoLocalRef(input: {
|
||||||
|
localDir: string;
|
||||||
|
bundlePath: string;
|
||||||
|
exportRef: string;
|
||||||
|
importedRef: string;
|
||||||
|
baseSha: string;
|
||||||
|
}): Promise<string> {
|
||||||
|
const bundleSize = (await fs.stat(input.bundlePath).catch(() => null))?.size ?? 0;
|
||||||
|
if (bundleSize === 0) {
|
||||||
|
return input.baseSha;
|
||||||
|
}
|
||||||
|
|
||||||
|
await runLocalGit(input.localDir, ["fetch", "--force", input.bundlePath, `${input.exportRef}:${input.importedRef}`], {
|
||||||
|
timeout: 60_000,
|
||||||
|
maxBuffer: 1024 * 1024,
|
||||||
|
});
|
||||||
|
const importedHead = await runLocalGit(input.localDir, ["rev-parse", input.importedRef], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
});
|
||||||
|
return importedHead.stdout.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
export function buildRemoteGitDeltaBundleScript(input: {
|
||||||
|
remoteDir: string;
|
||||||
|
baseSha: string;
|
||||||
|
exportRef: string;
|
||||||
|
bundlePath: string;
|
||||||
|
catBundle?: boolean;
|
||||||
|
cleanupBundle?: boolean;
|
||||||
|
}): string {
|
||||||
|
const remoteDir = shellQuote(input.remoteDir);
|
||||||
|
const bundlePath = shellQuote(input.bundlePath);
|
||||||
|
const exportRef = shellQuote(input.exportRef);
|
||||||
|
const baseSha = shellQuote(input.baseSha);
|
||||||
|
const cleanupParts = [
|
||||||
|
`rm -f ${bundlePath}`,
|
||||||
|
`git -C ${remoteDir} update-ref -d ${exportRef} >/dev/null 2>&1 || true`,
|
||||||
|
];
|
||||||
|
return [
|
||||||
|
"set -e",
|
||||||
|
input.cleanupBundle ? `cleanup() { ${cleanupParts.join("; ")}; }` : "",
|
||||||
|
input.cleanupBundle ? "trap cleanup EXIT" : "",
|
||||||
|
`mkdir -p ${shellQuote(path.posix.dirname(input.bundlePath))}`,
|
||||||
|
`rm -f ${bundlePath}`,
|
||||||
|
`git -C ${remoteDir} cat-file -e ${baseSha}^{commit}`,
|
||||||
|
`commit_count=$(git -C ${remoteDir} rev-list --count HEAD --not ${baseSha})`,
|
||||||
|
'if [ "$commit_count" -gt 0 ]; then',
|
||||||
|
` git -C ${remoteDir} update-ref ${exportRef} HEAD`,
|
||||||
|
` git -C ${remoteDir} bundle create ${bundlePath} ${exportRef} --not ${baseSha} >/dev/null`,
|
||||||
|
"else",
|
||||||
|
` : > ${bundlePath}`,
|
||||||
|
"fi",
|
||||||
|
input.catBundle ? `cat ${bundlePath}` : "",
|
||||||
|
].filter(Boolean).join("\n");
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function integrateImportedGitHead(input: {
|
||||||
|
localDir: string;
|
||||||
|
importedHead: string;
|
||||||
|
}): Promise<void> {
|
||||||
|
const isConcurrentRefUpdateError = (error: unknown) => {
|
||||||
|
const message = error instanceof Error ? error.message : String(error);
|
||||||
|
return message.includes("cannot lock ref") && message.includes("expected");
|
||||||
|
};
|
||||||
|
|
||||||
|
for (let attempt = 0; attempt < 5; attempt += 1) {
|
||||||
|
const snapshot = await readGitWorkspaceSnapshot(input.localDir);
|
||||||
|
if (!snapshot) return;
|
||||||
|
|
||||||
|
const currentHead = snapshot.headCommit;
|
||||||
|
if (!currentHead || currentHead === input.importedHead) return;
|
||||||
|
|
||||||
|
const headRef = snapshot.branchName ? `refs/heads/${snapshot.branchName}` : "HEAD";
|
||||||
|
const mergeBase = await runLocalGit(input.localDir, ["merge-base", currentHead, input.importedHead], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
}).catch(() => null);
|
||||||
|
const mergeBaseHead = mergeBase?.stdout.trim() ?? "";
|
||||||
|
|
||||||
|
if (mergeBaseHead === input.importedHead) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (mergeBaseHead === currentHead) {
|
||||||
|
try {
|
||||||
|
await runLocalGit(input.localDir, ["update-ref", headRef, input.importedHead, currentHead], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
} catch (error) {
|
||||||
|
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
let mergedTree;
|
||||||
|
try {
|
||||||
|
mergedTree = await runLocalGit(input.localDir, ["merge-tree", "--write-tree", currentHead, input.importedHead], {
|
||||||
|
timeout: 60_000,
|
||||||
|
maxBuffer: 256 * 1024,
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
const reason = error instanceof Error ? error.message : String(error);
|
||||||
|
throw new Error(
|
||||||
|
`Failed to merge concurrent remote git histories for ${currentHead.slice(0, 12)} and ${input.importedHead.slice(0, 12)}: ${reason}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const mergedTreeId = mergedTree.stdout.trim().split("\n")[0]?.trim() ?? "";
|
||||||
|
if (!mergedTreeId) {
|
||||||
|
throw new Error("Failed to compute a merged git tree for workspace restore.");
|
||||||
|
}
|
||||||
|
|
||||||
|
const mergeCommit = await runLocalGit(
|
||||||
|
input.localDir,
|
||||||
|
[
|
||||||
|
"commit-tree",
|
||||||
|
mergedTreeId,
|
||||||
|
"-p",
|
||||||
|
currentHead,
|
||||||
|
"-p",
|
||||||
|
input.importedHead,
|
||||||
|
"-m",
|
||||||
|
`Paperclip remote git sync merge ${input.importedHead.slice(0, 12)}`,
|
||||||
|
],
|
||||||
|
{
|
||||||
|
timeout: 60_000,
|
||||||
|
maxBuffer: 64 * 1024,
|
||||||
|
},
|
||||||
|
);
|
||||||
|
try {
|
||||||
|
await runLocalGit(input.localDir, ["update-ref", headRef, mergeCommit.stdout.trim(), currentHead], {
|
||||||
|
timeout: 10_000,
|
||||||
|
maxBuffer: 16 * 1024,
|
||||||
|
});
|
||||||
|
return;
|
||||||
|
} catch (error) {
|
||||||
|
if (isConcurrentRefUpdateError(error) && attempt < 4) continue;
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new Error(`Failed to integrate concurrent remote git history for ${input.importedHead.slice(0, 12)} after multiple retries.`);
|
||||||
|
}
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
import path from "node:path";
|
import path from "node:path";
|
||||||
|
import { GIT_ARCHIVE_EXCLUDES } from "./git-workspace-sync.js";
|
||||||
import {
|
import {
|
||||||
type SshRemoteExecutionSpec,
|
type SshRemoteExecutionSpec,
|
||||||
prepareWorkspaceForSshExecution,
|
prepareWorkspaceForSshExecution,
|
||||||
@@ -90,7 +91,7 @@ export async function prepareRemoteManagedRuntime(input: {
|
|||||||
remoteDir: workspaceRemoteDir,
|
remoteDir: workspaceRemoteDir,
|
||||||
onProgress: input.onProgress,
|
onProgress: input.onProgress,
|
||||||
});
|
});
|
||||||
const restoreExclude = preparedWorkspace.gitBacked ? [".git", ".paperclip-runtime"] : [".paperclip-runtime"];
|
const restoreExclude = preparedWorkspace.gitBacked ? [...GIT_ARCHIVE_EXCLUDES, ".paperclip-runtime"] : [".paperclip-runtime"];
|
||||||
const baselineSnapshot = await captureDirectorySnapshot(input.workspaceLocalDir, {
|
const baselineSnapshot = await captureDirectorySnapshot(input.workspaceLocalDir, {
|
||||||
exclude: restoreExclude,
|
exclude: restoreExclude,
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -13,6 +13,20 @@ import {
|
|||||||
|
|
||||||
const execFile = promisify(execFileCallback);
|
const execFile = promisify(execFileCallback);
|
||||||
|
|
||||||
|
async function git(cwd: string, args: string[]): Promise<string> {
|
||||||
|
const { stdout } = await execFile("git", ["-C", cwd, ...args], {
|
||||||
|
maxBuffer: 32 * 1024 * 1024,
|
||||||
|
});
|
||||||
|
return stdout.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function listTarMembers(rootDir: string, name: string, bytes: Buffer): Promise<string[]> {
|
||||||
|
const tarPath = path.join(rootDir, name);
|
||||||
|
await writeFile(tarPath, bytes);
|
||||||
|
const { stdout } = await execFile("tar", ["-tf", tarPath], { maxBuffer: 32 * 1024 * 1024 });
|
||||||
|
return stdout.split("\n").map((line) => line.trim()).filter(Boolean);
|
||||||
|
}
|
||||||
|
|
||||||
describe("sandbox managed runtime", () => {
|
describe("sandbox managed runtime", () => {
|
||||||
const cleanupDirs: string[] = [];
|
const cleanupDirs: string[] = [];
|
||||||
|
|
||||||
@@ -131,6 +145,224 @@ describe("sandbox managed runtime", () => {
|
|||||||
await expect(readFile(path.join(localWorkspaceDir, ".paperclip-runtime", "state.json"), "utf8")).resolves.toBe("{}\n");
|
await expect(readFile(path.join(localWorkspaceDir, ".paperclip-runtime", "state.json"), "utf8")).resolves.toBe("{}\n");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("syncs git-backed workspaces through a shallow standalone clone and keeps .git out of archives", async () => {
|
||||||
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-git-"));
|
||||||
|
cleanupDirs.push(rootDir);
|
||||||
|
const sourceRepoDir = path.join(rootDir, "source-repo");
|
||||||
|
const localWorkspaceDir = path.join(rootDir, "local-worktree");
|
||||||
|
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||||
|
|
||||||
|
await mkdir(sourceRepoDir, { recursive: true });
|
||||||
|
await git(sourceRepoDir, ["init"]);
|
||||||
|
await git(sourceRepoDir, ["checkout", "-b", "main"]);
|
||||||
|
await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]);
|
||||||
|
await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]);
|
||||||
|
await writeFile(path.join(sourceRepoDir, ".gitignore"), "node_modules/\n", "utf8");
|
||||||
|
await writeFile(path.join(sourceRepoDir, "tracked.txt"), "base\n", "utf8");
|
||||||
|
await writeFile(path.join(sourceRepoDir, "clean.txt"), "from git\n", "utf8");
|
||||||
|
await writeFile(path.join(sourceRepoDir, "deleted.txt"), "delete me\n", "utf8");
|
||||||
|
await git(sourceRepoDir, ["add", ".gitignore", "tracked.txt", "clean.txt", "deleted.txt"]);
|
||||||
|
await git(sourceRepoDir, ["commit", "-m", "base"]);
|
||||||
|
await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]);
|
||||||
|
|
||||||
|
expect((await lstat(path.join(localWorkspaceDir, ".git"))).isFile()).toBe(true);
|
||||||
|
await mkdir(path.join(localWorkspaceDir, "node_modules"), { recursive: true });
|
||||||
|
await writeFile(path.join(localWorkspaceDir, "tracked.txt"), "dirty local\n", "utf8");
|
||||||
|
await writeFile(path.join(localWorkspaceDir, "untracked.txt"), "from local\n", "utf8");
|
||||||
|
await writeFile(path.join(localWorkspaceDir, "node_modules", "cache.bin"), "do not upload\n", "utf8");
|
||||||
|
await rm(path.join(localWorkspaceDir, "deleted.txt"));
|
||||||
|
|
||||||
|
const uploadedTars: { remotePath: string; bytes: Buffer }[] = [];
|
||||||
|
const downloadedTars: { remotePath: string; bytes: Buffer }[] = [];
|
||||||
|
const client: SandboxManagedRuntimeClient = {
|
||||||
|
makeDir: async (remotePath) => {
|
||||||
|
await mkdir(remotePath, { recursive: true });
|
||||||
|
},
|
||||||
|
writeFile: async (remotePath, bytes) => {
|
||||||
|
await mkdir(path.dirname(remotePath), { recursive: true });
|
||||||
|
const buffer = Buffer.from(bytes);
|
||||||
|
if (remotePath.endsWith("-upload.tar")) uploadedTars.push({ remotePath, bytes: buffer });
|
||||||
|
await writeFile(remotePath, buffer);
|
||||||
|
},
|
||||||
|
readFile: async (remotePath) => {
|
||||||
|
const buffer = await readFile(remotePath);
|
||||||
|
if (remotePath.endsWith("workspace-download.tar")) downloadedTars.push({ remotePath, bytes: buffer });
|
||||||
|
return buffer;
|
||||||
|
},
|
||||||
|
listFiles: async () => [],
|
||||||
|
remove: async (remotePath) => {
|
||||||
|
await rm(remotePath, { recursive: true, force: true });
|
||||||
|
},
|
||||||
|
run: async (command) => {
|
||||||
|
await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 });
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const prepared = await prepareSandboxManagedRuntime({
|
||||||
|
spec: {
|
||||||
|
transport: "sandbox",
|
||||||
|
provider: "test",
|
||||||
|
sandboxId: "sandbox-1",
|
||||||
|
remoteCwd: remoteWorkspaceDir,
|
||||||
|
timeoutMs: 30_000,
|
||||||
|
apiKey: null,
|
||||||
|
},
|
||||||
|
adapterKey: "test-adapter",
|
||||||
|
client,
|
||||||
|
workspaceLocalDir: localWorkspaceDir,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect((await lstat(path.join(remoteWorkspaceDir, ".git"))).isDirectory()).toBe(true);
|
||||||
|
await expect(readFile(path.join(remoteWorkspaceDir, ".git", "shallow"), "utf8")).resolves.toContain(
|
||||||
|
await git(localWorkspaceDir, ["rev-parse", "HEAD"]),
|
||||||
|
);
|
||||||
|
expect(await git(remoteWorkspaceDir, ["rev-list", "--count", "HEAD"])).toBe("1");
|
||||||
|
expect(await git(remoteWorkspaceDir, ["status", "--short"])).toContain("M tracked.txt");
|
||||||
|
expect(await git(remoteWorkspaceDir, ["status", "--short"])).toContain("?? untracked.txt");
|
||||||
|
await expect(readFile(path.join(remoteWorkspaceDir, "deleted.txt"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||||
|
|
||||||
|
const gitUpload = uploadedTars.find((entry) => path.posix.basename(entry.remotePath) === "git-workspace-upload.tar");
|
||||||
|
const workspaceUpload = uploadedTars.find((entry) => path.posix.basename(entry.remotePath) === "workspace-upload.tar");
|
||||||
|
expect(gitUpload).toBeDefined();
|
||||||
|
expect(workspaceUpload).toBeDefined();
|
||||||
|
const gitMembers = await listTarMembers(rootDir, "git-upload-list.tar", gitUpload!.bytes);
|
||||||
|
const workspaceMembers = await listTarMembers(rootDir, "workspace-upload-list.tar", workspaceUpload!.bytes);
|
||||||
|
expect(gitMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(true);
|
||||||
|
expect(workspaceMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(false);
|
||||||
|
expect(workspaceMembers).toContain("tracked.txt");
|
||||||
|
expect(workspaceMembers).toContain("untracked.txt");
|
||||||
|
expect(workspaceMembers).not.toContain("clean.txt");
|
||||||
|
expect(workspaceMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
|
||||||
|
|
||||||
|
await git(remoteWorkspaceDir, ["config", "user.name", "Paperclip Sandbox"]);
|
||||||
|
await git(remoteWorkspaceDir, ["config", "user.email", "sandbox@paperclip.dev"]);
|
||||||
|
await git(remoteWorkspaceDir, ["add", "-A"]);
|
||||||
|
await git(remoteWorkspaceDir, ["commit", "-m", "sandbox update"]);
|
||||||
|
await writeFile(path.join(remoteWorkspaceDir, "tracked.txt"), "remote dirty\n", "utf8");
|
||||||
|
await writeFile(path.join(remoteWorkspaceDir, "remote-only.txt"), "from sandbox\n", "utf8");
|
||||||
|
|
||||||
|
await prepared.restoreWorkspace();
|
||||||
|
|
||||||
|
expect((await lstat(path.join(localWorkspaceDir, ".git"))).isFile()).toBe(true);
|
||||||
|
expect(await git(localWorkspaceDir, ["log", "-1", "--pretty=%s"])).toBe("sandbox update");
|
||||||
|
await expect(readFile(path.join(localWorkspaceDir, "tracked.txt"), "utf8")).resolves.toBe("remote dirty\n");
|
||||||
|
await expect(readFile(path.join(localWorkspaceDir, "remote-only.txt"), "utf8")).resolves.toBe("from sandbox\n");
|
||||||
|
await expect(readFile(path.join(localWorkspaceDir, "deleted.txt"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||||
|
await expect(readFile(path.join(localWorkspaceDir, "node_modules", "cache.bin"), "utf8")).resolves.toBe("do not upload\n");
|
||||||
|
|
||||||
|
expect(downloadedTars).toHaveLength(1);
|
||||||
|
const downloadMembers = await listTarMembers(rootDir, "workspace-download-list.tar", downloadedTars[0]!.bytes);
|
||||||
|
expect(downloadMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(false);
|
||||||
|
expect(downloadMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("excludes unignored dependency trees from git-backed workspace overlay archives", async () => {
|
||||||
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-unignored-deps-"));
|
||||||
|
cleanupDirs.push(rootDir);
|
||||||
|
const sourceRepoDir = path.join(rootDir, "source-repo");
|
||||||
|
const localWorkspaceDir = path.join(rootDir, "local-worktree");
|
||||||
|
const remoteWorkspaceDir = path.join(rootDir, "remote-workspace");
|
||||||
|
|
||||||
|
await mkdir(sourceRepoDir, { recursive: true });
|
||||||
|
await git(sourceRepoDir, ["init"]);
|
||||||
|
await git(sourceRepoDir, ["checkout", "-b", "main"]);
|
||||||
|
await git(sourceRepoDir, ["config", "user.name", "Paperclip Test"]);
|
||||||
|
await git(sourceRepoDir, ["config", "user.email", "test@paperclip.dev"]);
|
||||||
|
await mkdir(path.join(sourceRepoDir, "src"), { recursive: true });
|
||||||
|
await writeFile(path.join(sourceRepoDir, "src", "tracked.ts"), "export const tracked = true;\n", "utf8");
|
||||||
|
await git(sourceRepoDir, ["add", "src/tracked.ts"]);
|
||||||
|
await git(sourceRepoDir, ["commit", "-m", "base"]);
|
||||||
|
await git(sourceRepoDir, ["worktree", "add", "-b", "work", localWorkspaceDir, "HEAD"]);
|
||||||
|
|
||||||
|
await mkdir(path.join(localWorkspaceDir, "node_modules", "root-package"), { recursive: true });
|
||||||
|
await mkdir(path.join(localWorkspaceDir, "packages", "ui", "node_modules", "nested-package"), { recursive: true });
|
||||||
|
await writeFile(path.join(localWorkspaceDir, "node_modules", "root-package", "cache.bin"), "root dependency\n", "utf8");
|
||||||
|
await writeFile(
|
||||||
|
path.join(localWorkspaceDir, "packages", "ui", "node_modules", "nested-package", "cache.bin"),
|
||||||
|
"nested dependency\n",
|
||||||
|
"utf8",
|
||||||
|
);
|
||||||
|
await writeFile(path.join(localWorkspaceDir, "src", "local-only.ts"), "export const local = true;\n", "utf8");
|
||||||
|
|
||||||
|
const uploadedTars: { remotePath: string; bytes: Buffer }[] = [];
|
||||||
|
const downloadedTars: { remotePath: string; bytes: Buffer }[] = [];
|
||||||
|
const client: SandboxManagedRuntimeClient = {
|
||||||
|
makeDir: async (remotePath) => {
|
||||||
|
await mkdir(remotePath, { recursive: true });
|
||||||
|
},
|
||||||
|
writeFile: async (remotePath, bytes) => {
|
||||||
|
await mkdir(path.dirname(remotePath), { recursive: true });
|
||||||
|
const buffer = Buffer.from(bytes);
|
||||||
|
if (remotePath.endsWith("-upload.tar")) uploadedTars.push({ remotePath, bytes: buffer });
|
||||||
|
await writeFile(remotePath, buffer);
|
||||||
|
},
|
||||||
|
readFile: async (remotePath) => {
|
||||||
|
const buffer = await readFile(remotePath);
|
||||||
|
if (remotePath.endsWith("workspace-download.tar")) downloadedTars.push({ remotePath, bytes: buffer });
|
||||||
|
return buffer;
|
||||||
|
},
|
||||||
|
listFiles: async () => [],
|
||||||
|
remove: async (remotePath) => {
|
||||||
|
await rm(remotePath, { recursive: true, force: true });
|
||||||
|
},
|
||||||
|
run: async (command) => {
|
||||||
|
await execFile("sh", ["-c", command], { maxBuffer: 32 * 1024 * 1024 });
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const prepared = await prepareSandboxManagedRuntime({
|
||||||
|
spec: {
|
||||||
|
transport: "sandbox",
|
||||||
|
provider: "test",
|
||||||
|
sandboxId: "sandbox-1",
|
||||||
|
remoteCwd: remoteWorkspaceDir,
|
||||||
|
timeoutMs: 30_000,
|
||||||
|
apiKey: null,
|
||||||
|
},
|
||||||
|
adapterKey: "test-adapter",
|
||||||
|
client,
|
||||||
|
workspaceLocalDir: localWorkspaceDir,
|
||||||
|
});
|
||||||
|
|
||||||
|
const workspaceUpload = uploadedTars.find((entry) => path.posix.basename(entry.remotePath) === "workspace-upload.tar");
|
||||||
|
expect(workspaceUpload).toBeDefined();
|
||||||
|
const workspaceMembers = await listTarMembers(rootDir, "unignored-deps-workspace-upload.tar", workspaceUpload!.bytes);
|
||||||
|
expect(workspaceMembers).toContain("src/local-only.ts");
|
||||||
|
expect(workspaceMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
|
||||||
|
expect(workspaceMembers.some((entry) => entry.includes("/node_modules/") || entry.endsWith("/node_modules"))).toBe(false);
|
||||||
|
|
||||||
|
await expect(readFile(path.join(remoteWorkspaceDir, "src", "local-only.ts"), "utf8")).resolves.toBe("export const local = true;\n");
|
||||||
|
await expect(readFile(path.join(remoteWorkspaceDir, "node_modules", "root-package", "cache.bin"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||||
|
await expect(
|
||||||
|
readFile(path.join(remoteWorkspaceDir, "packages", "ui", "node_modules", "nested-package", "cache.bin"), "utf8"),
|
||||||
|
).rejects.toMatchObject({ code: "ENOENT" });
|
||||||
|
|
||||||
|
await mkdir(path.join(remoteWorkspaceDir, "node_modules", "sandbox-package"), { recursive: true });
|
||||||
|
await mkdir(path.join(remoteWorkspaceDir, "packages", "ui", "node_modules", "sandbox-package"), { recursive: true });
|
||||||
|
await writeFile(path.join(remoteWorkspaceDir, "node_modules", "sandbox-package", "cache.bin"), "sandbox root dependency\n", "utf8");
|
||||||
|
await writeFile(
|
||||||
|
path.join(remoteWorkspaceDir, "packages", "ui", "node_modules", "sandbox-package", "cache.bin"),
|
||||||
|
"sandbox nested dependency\n",
|
||||||
|
"utf8",
|
||||||
|
);
|
||||||
|
await writeFile(path.join(remoteWorkspaceDir, "src", "remote-only.ts"), "export const remote = true;\n", "utf8");
|
||||||
|
|
||||||
|
await prepared.restoreWorkspace();
|
||||||
|
|
||||||
|
await expect(readFile(path.join(localWorkspaceDir, "node_modules", "root-package", "cache.bin"), "utf8")).resolves.toBe("root dependency\n");
|
||||||
|
await expect(
|
||||||
|
readFile(path.join(localWorkspaceDir, "packages", "ui", "node_modules", "nested-package", "cache.bin"), "utf8"),
|
||||||
|
).resolves.toBe("nested dependency\n");
|
||||||
|
await expect(readFile(path.join(localWorkspaceDir, "src", "remote-only.ts"), "utf8")).resolves.toBe("export const remote = true;\n");
|
||||||
|
await expect(readFile(path.join(localWorkspaceDir, "node_modules", "sandbox-package", "cache.bin"), "utf8")).rejects.toMatchObject({ code: "ENOENT" });
|
||||||
|
|
||||||
|
expect(downloadedTars).toHaveLength(1);
|
||||||
|
const downloadMembers = await listTarMembers(rootDir, "unignored-deps-workspace-download.tar", downloadedTars[0]!.bytes);
|
||||||
|
expect(downloadMembers.some((entry) => entry === ".git" || entry.startsWith(".git/"))).toBe(false);
|
||||||
|
expect(downloadMembers.some((entry) => entry === "node_modules" || entry.startsWith("node_modules/"))).toBe(false);
|
||||||
|
expect(downloadMembers.some((entry) => entry.includes("/node_modules/") || entry.endsWith("/node_modules"))).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
it("builds workspace/asset tarballs without a './' self-entry (so untar does not chmod/utime an unowned target dir)", async () => {
|
it("builds workspace/asset tarballs without a './' self-entry (so untar does not chmod/utime an unowned target dir)", async () => {
|
||||||
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-tarself-"));
|
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-tarself-"));
|
||||||
cleanupDirs.push(rootDir);
|
cleanupDirs.push(rootDir);
|
||||||
|
|||||||
@@ -3,10 +3,44 @@ import { constants as fsConstants, promises as fs } from "node:fs";
|
|||||||
import os from "node:os";
|
import os from "node:os";
|
||||||
import path from "node:path";
|
import path from "node:path";
|
||||||
import { promisify } from "node:util";
|
import { promisify } from "node:util";
|
||||||
|
import {
|
||||||
|
buildRemoteGitDeltaBundleScript,
|
||||||
|
createImportedGitRef,
|
||||||
|
createRemoteGitExportRef,
|
||||||
|
deleteLocalGitRef,
|
||||||
|
fetchGitBundleIntoLocalRef,
|
||||||
|
GIT_ARCHIVE_EXCLUDES,
|
||||||
|
integrateImportedGitHead,
|
||||||
|
readGitWorkspaceSnapshot,
|
||||||
|
withShallowGitWorkspaceClone,
|
||||||
|
} from "./git-workspace-sync.js";
|
||||||
import { captureDirectorySnapshot, mergeDirectoryWithBaseline } from "./workspace-restore-merge.js";
|
import { captureDirectorySnapshot, mergeDirectoryWithBaseline } from "./workspace-restore-merge.js";
|
||||||
import { createRuntimeProgressReporter, type RuntimeProgressSink } from "./runtime-progress.js";
|
import {
|
||||||
|
createRuntimeProgressReporter,
|
||||||
|
type RuntimeProgressDirection,
|
||||||
|
type RuntimeProgressPhase,
|
||||||
|
type RuntimeProgressSink,
|
||||||
|
} from "./runtime-progress.js";
|
||||||
|
import { isRelativePathOrDescendant, shouldExcludePath } from "./exclude-patterns.js";
|
||||||
|
|
||||||
const execFile = promisify(execFileCallback);
|
const execFile = promisify(execFileCallback);
|
||||||
|
const SANDBOX_WORKSPACE_HEAVY_DIR_NAMES = [
|
||||||
|
"node_modules",
|
||||||
|
"vendor",
|
||||||
|
"dist",
|
||||||
|
"build",
|
||||||
|
"out",
|
||||||
|
"coverage",
|
||||||
|
".next",
|
||||||
|
".turbo",
|
||||||
|
".cache",
|
||||||
|
] as const;
|
||||||
|
const SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES = SANDBOX_WORKSPACE_HEAVY_DIR_NAMES.flatMap((entry) => [
|
||||||
|
entry,
|
||||||
|
`${entry}/*`,
|
||||||
|
`*/${entry}`,
|
||||||
|
`*/${entry}/*`,
|
||||||
|
]);
|
||||||
|
|
||||||
export interface SandboxRemoteExecutionSpec {
|
export interface SandboxRemoteExecutionSpec {
|
||||||
transport: "sandbox";
|
transport: "sandbox";
|
||||||
@@ -208,8 +242,28 @@ async function walkDirectory(root: string, relative = ""): Promise<string[]> {
|
|||||||
return out.sort((left, right) => right.length - left.length);
|
return out.sort((left, right) => right.length - left.length);
|
||||||
}
|
}
|
||||||
|
|
||||||
function isRelativePathOrDescendant(relative: string, candidate: string): boolean {
|
async function copyWorkspaceEntry(sourceRoot: string, targetRoot: string, relative: string): Promise<void> {
|
||||||
return relative === candidate || relative.startsWith(`${candidate}/`);
|
const sourcePath = path.join(sourceRoot, relative);
|
||||||
|
const targetPath = path.join(targetRoot, relative);
|
||||||
|
const stats = await fs.lstat(sourcePath);
|
||||||
|
|
||||||
|
if (stats.isDirectory()) {
|
||||||
|
await fs.mkdir(targetPath, { recursive: true });
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
await fs.mkdir(path.dirname(targetPath), { recursive: true });
|
||||||
|
await fs.rm(targetPath, { recursive: true, force: true }).catch(() => undefined);
|
||||||
|
if (stats.isSymbolicLink()) {
|
||||||
|
const linkTarget = await fs.readlink(sourcePath);
|
||||||
|
await fs.symlink(linkTarget, targetPath);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
await fs.copyFile(sourcePath, targetPath, fsConstants.COPYFILE_FICLONE).catch(async () => {
|
||||||
|
await fs.copyFile(sourcePath, targetPath);
|
||||||
|
});
|
||||||
|
await fs.chmod(targetPath, stats.mode);
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function mirrorDirectory(
|
export async function mirrorDirectory(
|
||||||
@@ -231,33 +285,24 @@ export async function mirrorDirectory(
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
const copyEntry = async (relative: string) => {
|
|
||||||
const sourcePath = path.join(sourceDir, relative);
|
|
||||||
const targetPath = path.join(targetDir, relative);
|
|
||||||
const stats = await fs.lstat(sourcePath);
|
|
||||||
|
|
||||||
if (stats.isDirectory()) {
|
|
||||||
await fs.mkdir(targetPath, { recursive: true });
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
await fs.mkdir(path.dirname(targetPath), { recursive: true });
|
|
||||||
await fs.rm(targetPath, { recursive: true, force: true }).catch(() => undefined);
|
|
||||||
if (stats.isSymbolicLink()) {
|
|
||||||
const linkTarget = await fs.readlink(sourcePath);
|
|
||||||
await fs.symlink(linkTarget, targetPath);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
await fs.copyFile(sourcePath, targetPath, fsConstants.COPYFILE_FICLONE).catch(async () => {
|
|
||||||
await fs.copyFile(sourcePath, targetPath);
|
|
||||||
});
|
|
||||||
await fs.chmod(targetPath, stats.mode);
|
|
||||||
};
|
|
||||||
|
|
||||||
const entries = (await walkDirectory(sourceDir)).sort((left, right) => left.localeCompare(right));
|
const entries = (await walkDirectory(sourceDir)).sort((left, right) => left.localeCompare(right));
|
||||||
for (const relative of entries) {
|
for (const relative of entries) {
|
||||||
await copyEntry(relative);
|
await copyWorkspaceEntry(sourceDir, targetDir, relative);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function copySelectedWorkspaceEntries(input: {
|
||||||
|
sourceDir: string;
|
||||||
|
targetDir: string;
|
||||||
|
relativePaths: string[];
|
||||||
|
exclude: string[];
|
||||||
|
}): Promise<void> {
|
||||||
|
await fs.mkdir(input.targetDir, { recursive: true });
|
||||||
|
for (const relative of input.relativePaths) {
|
||||||
|
if (shouldExcludePath(relative, input.exclude)) continue;
|
||||||
|
const sourceStats = await fs.lstat(path.join(input.sourceDir, relative)).catch(() => null);
|
||||||
|
if (!sourceStats) continue;
|
||||||
|
await copyWorkspaceEntry(input.sourceDir, input.targetDir, relative);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -275,15 +320,37 @@ function tarExcludeFlags(exclude: string[] | undefined): string {
|
|||||||
return ["._*", ...(exclude ?? [])].map((entry) => `--exclude ${shellQuote(entry)}`).join(" ");
|
return ["._*", ...(exclude ?? [])].map((entry) => `--exclude ${shellQuote(entry)}`).join(" ");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function mergeExcludes(...groups: Array<string[] | undefined>): string[] {
|
||||||
|
return [...new Set(groups.flatMap((group) => group ?? []))];
|
||||||
|
}
|
||||||
|
|
||||||
|
function preserveFindArgs(entries: string[]): string {
|
||||||
|
return entries.map((entry) => `! -name ${shellQuote(entry)}`).join(" ");
|
||||||
|
}
|
||||||
|
|
||||||
|
async function removeDeletedPathsInSandbox(input: {
|
||||||
|
client: SandboxManagedRuntimeClient;
|
||||||
|
spec: SandboxRemoteExecutionSpec;
|
||||||
|
remoteDir: string;
|
||||||
|
deletedPaths: string[];
|
||||||
|
}): Promise<void> {
|
||||||
|
if (input.deletedPaths.length === 0) return;
|
||||||
|
const quotedPaths = input.deletedPaths.map((entry) => shellQuote(entry)).join(" ");
|
||||||
|
await input.client.run(
|
||||||
|
`sh -c ${shellQuote(`cd ${shellQuote(input.remoteDir)} && rm -rf -- ${quotedPaths}`)}`,
|
||||||
|
{ timeoutMs: input.spec.timeoutMs },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
// Bridge a single byte-level transfer to the throttled progress reporter. The
|
// Bridge a single byte-level transfer to the throttled progress reporter. The
|
||||||
// transport reports decoded bytes via `options.onProgress`; the reporter turns
|
// transport reports decoded bytes via `options.onProgress`; the reporter turns
|
||||||
// them into a throttled, fully-formatted log line. `finish()` emits the terminal
|
// them into a throttled, fully-formatted log line. `finish()` emits the terminal
|
||||||
// completion line (idempotent) once the transfer returns.
|
// completion line (idempotent) once the transfer returns.
|
||||||
function makeTransferProgress(
|
function makeTransferProgress(
|
||||||
sink: RuntimeProgressSink | undefined,
|
sink: RuntimeProgressSink | undefined,
|
||||||
phase: "Syncing" | "Restoring",
|
phase: RuntimeProgressPhase,
|
||||||
direction: "to" | "from",
|
direction: RuntimeProgressDirection,
|
||||||
label: string,
|
label?: string,
|
||||||
): { options: SandboxTransferProgressOptions | undefined; finish: () => Promise<void> } {
|
): { options: SandboxTransferProgressOptions | undefined; finish: () => Promise<void> } {
|
||||||
if (!sink) return { options: undefined, finish: async () => {} };
|
if (!sink) return { options: undefined, finish: async () => {} };
|
||||||
const reporter = createRuntimeProgressReporter({
|
const reporter = createRuntimeProgressReporter({
|
||||||
@@ -320,16 +387,75 @@ export async function prepareSandboxManagedRuntime(input: {
|
|||||||
}): Promise<PreparedSandboxManagedRuntime> {
|
}): Promise<PreparedSandboxManagedRuntime> {
|
||||||
const workspaceRemoteDir = input.workspaceRemoteDir ?? input.spec.remoteCwd;
|
const workspaceRemoteDir = input.workspaceRemoteDir ?? input.spec.remoteCwd;
|
||||||
const runtimeRootDir = path.posix.join(workspaceRemoteDir, ".paperclip-runtime", input.adapterKey);
|
const runtimeRootDir = path.posix.join(workspaceRemoteDir, ".paperclip-runtime", input.adapterKey);
|
||||||
|
const gitSnapshot = await readGitWorkspaceSnapshot(input.workspaceLocalDir);
|
||||||
|
const gitIgnoredExcludes = gitSnapshot?.ignoredPaths;
|
||||||
|
const workspaceArchiveExclude = mergeExcludes(
|
||||||
|
SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES,
|
||||||
|
[...GIT_ARCHIVE_EXCLUDES],
|
||||||
|
input.workspaceExclude,
|
||||||
|
gitIgnoredExcludes,
|
||||||
|
);
|
||||||
|
const restoreExclude = mergeExcludes(
|
||||||
|
SANDBOX_WORKSPACE_HEAVY_DIR_EXCLUDES,
|
||||||
|
[...GIT_ARCHIVE_EXCLUDES],
|
||||||
|
[".paperclip-runtime"],
|
||||||
|
input.preserveAbsentOnRestore,
|
||||||
|
input.workspaceExclude,
|
||||||
|
gitIgnoredExcludes,
|
||||||
|
);
|
||||||
const baselineSnapshot = await captureDirectorySnapshot(input.workspaceLocalDir, {
|
const baselineSnapshot = await captureDirectorySnapshot(input.workspaceLocalDir, {
|
||||||
exclude: [...new Set([".paperclip-runtime", ...(input.preserveAbsentOnRestore ?? []), ...(input.workspaceExclude ?? [])])],
|
exclude: restoreExclude,
|
||||||
});
|
});
|
||||||
|
|
||||||
await withTempDir("paperclip-sandbox-sync-", async (tempDir) => {
|
await withTempDir("paperclip-sandbox-sync-", async (tempDir) => {
|
||||||
|
const preservedNames = new Set([
|
||||||
|
".paperclip-runtime",
|
||||||
|
...(gitSnapshot ? [".git"] : []),
|
||||||
|
...(input.preserveAbsentOnRestore ?? []),
|
||||||
|
]);
|
||||||
|
if (gitSnapshot) {
|
||||||
|
await withShallowGitWorkspaceClone({
|
||||||
|
localDir: input.workspaceLocalDir,
|
||||||
|
snapshot: gitSnapshot,
|
||||||
|
}, async (cloneDir) => {
|
||||||
|
const gitTarPath = path.join(tempDir, "git-workspace.tar");
|
||||||
|
await createTarballFromDirectory({
|
||||||
|
localDir: cloneDir,
|
||||||
|
archivePath: gitTarPath,
|
||||||
|
exclude: [".paperclip-runtime"],
|
||||||
|
});
|
||||||
|
const gitTarBytes = await fs.readFile(gitTarPath);
|
||||||
|
const remoteGitTar = path.posix.join(runtimeRootDir, "git-workspace-upload.tar");
|
||||||
|
await input.client.makeDir(runtimeRootDir);
|
||||||
|
const gitUpload = makeTransferProgress(input.onProgress, "Syncing", "to", "git history");
|
||||||
|
await input.client.writeFile(remoteGitTar, toArrayBuffer(gitTarBytes), gitUpload.options);
|
||||||
|
await gitUpload.finish();
|
||||||
|
await input.client.run(
|
||||||
|
`sh -c ${shellQuote(
|
||||||
|
`mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
|
||||||
|
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${preserveFindArgs([".paperclip-runtime"])} -exec rm -rf -- {} + && ` +
|
||||||
|
`tar -xf ${shellQuote(remoteGitTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
|
||||||
|
`rm -f ${shellQuote(remoteGitTar)}`,
|
||||||
|
)}`,
|
||||||
|
{ timeoutMs: input.spec.timeoutMs },
|
||||||
|
);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
const workspaceTarPath = path.join(tempDir, "workspace.tar");
|
const workspaceTarPath = path.join(tempDir, "workspace.tar");
|
||||||
|
const workspaceArchiveDir = gitSnapshot ? path.join(tempDir, "workspace-overlay") : input.workspaceLocalDir;
|
||||||
|
if (gitSnapshot) {
|
||||||
|
await copySelectedWorkspaceEntries({
|
||||||
|
sourceDir: input.workspaceLocalDir,
|
||||||
|
targetDir: workspaceArchiveDir,
|
||||||
|
relativePaths: gitSnapshot.overlayPaths,
|
||||||
|
exclude: workspaceArchiveExclude,
|
||||||
|
});
|
||||||
|
}
|
||||||
await createTarballFromDirectory({
|
await createTarballFromDirectory({
|
||||||
localDir: input.workspaceLocalDir,
|
localDir: workspaceArchiveDir,
|
||||||
archivePath: workspaceTarPath,
|
archivePath: workspaceTarPath,
|
||||||
exclude: input.workspaceExclude,
|
exclude: gitSnapshot ? undefined : workspaceArchiveExclude,
|
||||||
});
|
});
|
||||||
const workspaceTarBytes = await fs.readFile(workspaceTarPath);
|
const workspaceTarBytes = await fs.readFile(workspaceTarPath);
|
||||||
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-upload.tar");
|
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-upload.tar");
|
||||||
@@ -341,17 +467,26 @@ export async function prepareSandboxManagedRuntime(input: {
|
|||||||
workspaceUpload.options,
|
workspaceUpload.options,
|
||||||
);
|
);
|
||||||
await workspaceUpload.finish();
|
await workspaceUpload.finish();
|
||||||
const preservedNames = new Set([".paperclip-runtime", ...(input.preserveAbsentOnRestore ?? [])]);
|
const extractWorkspaceTarCommand = gitSnapshot
|
||||||
const findPreserveArgs = [...preservedNames].map((entry) => `! -name ${shellQuote(entry)}`).join(" ");
|
? `mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
|
||||||
|
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
|
||||||
|
`rm -f ${shellQuote(remoteWorkspaceTar)}`
|
||||||
|
: `mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
|
||||||
|
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${preserveFindArgs([...preservedNames])} -exec rm -rf -- {} + && ` +
|
||||||
|
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
|
||||||
|
`rm -f ${shellQuote(remoteWorkspaceTar)}`;
|
||||||
await input.client.run(
|
await input.client.run(
|
||||||
`sh -c ${shellQuote(
|
`sh -c ${shellQuote(extractWorkspaceTarCommand)}`,
|
||||||
`mkdir -p ${shellQuote(workspaceRemoteDir)} && ` +
|
|
||||||
`find ${shellQuote(workspaceRemoteDir)} -mindepth 1 -maxdepth 1 ${findPreserveArgs} -exec rm -rf -- {} + && ` +
|
|
||||||
`tar -xf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} && ` +
|
|
||||||
`rm -f ${shellQuote(remoteWorkspaceTar)}`,
|
|
||||||
)}`,
|
|
||||||
{ timeoutMs: input.spec.timeoutMs },
|
{ timeoutMs: input.spec.timeoutMs },
|
||||||
);
|
);
|
||||||
|
if (gitSnapshot) {
|
||||||
|
await removeDeletedPathsInSandbox({
|
||||||
|
client: input.client,
|
||||||
|
spec: input.spec,
|
||||||
|
remoteDir: workspaceRemoteDir,
|
||||||
|
deletedPaths: gitSnapshot.deletedPaths,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
for (const asset of input.assets ?? []) {
|
for (const asset of input.assets ?? []) {
|
||||||
const assetTarPath = path.join(tempDir, `${asset.key}.tar`);
|
const assetTarPath = path.join(tempDir, `${asset.key}.tar`);
|
||||||
@@ -392,31 +527,76 @@ export async function prepareSandboxManagedRuntime(input: {
|
|||||||
restoreWorkspace: async (onProgress?: RuntimeProgressSink) => {
|
restoreWorkspace: async (onProgress?: RuntimeProgressSink) => {
|
||||||
const restoreSink = onProgress ?? input.onProgress;
|
const restoreSink = onProgress ?? input.onProgress;
|
||||||
await withTempDir("paperclip-sandbox-restore-", async (tempDir) => {
|
await withTempDir("paperclip-sandbox-restore-", async (tempDir) => {
|
||||||
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-download.tar");
|
let importedRef: string | null = null;
|
||||||
await input.client.run(
|
let importedHead: string | null = null;
|
||||||
`sh -c ${shellQuote(
|
try {
|
||||||
`mkdir -p ${shellQuote(runtimeRootDir)} && ` +
|
if (gitSnapshot) {
|
||||||
`tar -cf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} ` +
|
importedRef = createImportedGitRef("sandbox");
|
||||||
`${tarExcludeFlags(input.workspaceExclude)} .`,
|
const remoteGitBundle = path.posix.join(runtimeRootDir, "git-delta.bundle");
|
||||||
)}`,
|
const exportRef = createRemoteGitExportRef("sandbox");
|
||||||
{ timeoutMs: input.spec.timeoutMs },
|
await input.client.run(
|
||||||
);
|
`sh -c ${shellQuote(buildRemoteGitDeltaBundleScript({
|
||||||
const workspaceRestore = makeTransferProgress(restoreSink, "Restoring", "from", "workspace");
|
remoteDir: workspaceRemoteDir,
|
||||||
const archiveBytes = await input.client.readFile(remoteWorkspaceTar, workspaceRestore.options);
|
baseSha: gitSnapshot.headCommit,
|
||||||
await workspaceRestore.finish();
|
exportRef,
|
||||||
await input.client.remove(remoteWorkspaceTar).catch(() => undefined);
|
bundlePath: remoteGitBundle,
|
||||||
const localArchivePath = path.join(tempDir, "workspace.tar");
|
}))}`,
|
||||||
const extractedDir = path.join(tempDir, "workspace");
|
{ timeoutMs: input.spec.timeoutMs },
|
||||||
await fs.writeFile(localArchivePath, toBuffer(archiveBytes));
|
);
|
||||||
await extractTarballToDirectory({
|
const gitExport = makeTransferProgress(restoreSink, "Exporting git history", "from");
|
||||||
archivePath: localArchivePath,
|
const bundleBytes = await input.client.readFile(remoteGitBundle, gitExport.options);
|
||||||
localDir: extractedDir,
|
await gitExport.finish();
|
||||||
});
|
await input.client.remove(remoteGitBundle).catch(() => undefined);
|
||||||
await mergeDirectoryWithBaseline({
|
const bundlePath = path.join(tempDir, "git-delta.bundle");
|
||||||
baseline: baselineSnapshot,
|
await fs.writeFile(bundlePath, toBuffer(bundleBytes));
|
||||||
sourceDir: extractedDir,
|
importedHead = await fetchGitBundleIntoLocalRef({
|
||||||
targetDir: input.workspaceLocalDir,
|
localDir: input.workspaceLocalDir,
|
||||||
});
|
bundlePath,
|
||||||
|
exportRef,
|
||||||
|
importedRef,
|
||||||
|
baseSha: gitSnapshot.headCommit,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
const remoteWorkspaceTar = path.posix.join(runtimeRootDir, "workspace-download.tar");
|
||||||
|
await input.client.run(
|
||||||
|
`sh -c ${shellQuote(
|
||||||
|
`mkdir -p ${shellQuote(runtimeRootDir)} && ` +
|
||||||
|
`tar -cf ${shellQuote(remoteWorkspaceTar)} -C ${shellQuote(workspaceRemoteDir)} ` +
|
||||||
|
`${tarExcludeFlags(restoreExclude)} .`,
|
||||||
|
)}`,
|
||||||
|
{ timeoutMs: input.spec.timeoutMs },
|
||||||
|
);
|
||||||
|
const workspaceRestore = makeTransferProgress(restoreSink, "Restoring", "from", "workspace");
|
||||||
|
const archiveBytes = await input.client.readFile(remoteWorkspaceTar, workspaceRestore.options);
|
||||||
|
await workspaceRestore.finish();
|
||||||
|
await input.client.remove(remoteWorkspaceTar).catch(() => undefined);
|
||||||
|
const localArchivePath = path.join(tempDir, "workspace.tar");
|
||||||
|
const extractedDir = path.join(tempDir, "workspace");
|
||||||
|
await fs.writeFile(localArchivePath, toBuffer(archiveBytes));
|
||||||
|
await extractTarballToDirectory({
|
||||||
|
archivePath: localArchivePath,
|
||||||
|
localDir: extractedDir,
|
||||||
|
});
|
||||||
|
const gitHeadToIntegrate = importedHead;
|
||||||
|
await mergeDirectoryWithBaseline({
|
||||||
|
baseline: baselineSnapshot,
|
||||||
|
sourceDir: extractedDir,
|
||||||
|
targetDir: input.workspaceLocalDir,
|
||||||
|
beforeApply: gitHeadToIntegrate
|
||||||
|
? async () => {
|
||||||
|
await integrateImportedGitHead({
|
||||||
|
localDir: input.workspaceLocalDir,
|
||||||
|
importedHead: gitHeadToIntegrate,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
: undefined,
|
||||||
|
});
|
||||||
|
} finally {
|
||||||
|
if (importedRef) {
|
||||||
|
await deleteLocalGitRef({ localDir: input.workspaceLocalDir, ref: importedRef });
|
||||||
|
}
|
||||||
|
}
|
||||||
});
|
});
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ import { createHash } from "node:crypto";
|
|||||||
import { createReadStream } from "node:fs";
|
import { createReadStream } from "node:fs";
|
||||||
import { constants as fsConstants, promises as fs } from "node:fs";
|
import { constants as fsConstants, promises as fs } from "node:fs";
|
||||||
import path from "node:path";
|
import path from "node:path";
|
||||||
|
import { shouldExcludePath } from "./exclude-patterns.js";
|
||||||
|
|
||||||
type SnapshotEntry =
|
type SnapshotEntry =
|
||||||
| { kind: "dir" }
|
| { kind: "dir" }
|
||||||
@@ -13,14 +14,6 @@ export interface DirectorySnapshot {
|
|||||||
entries: Map<string, SnapshotEntry>;
|
entries: Map<string, SnapshotEntry>;
|
||||||
}
|
}
|
||||||
|
|
||||||
function isRelativePathOrDescendant(relative: string, candidate: string): boolean {
|
|
||||||
return relative === candidate || relative.startsWith(`${candidate}/`);
|
|
||||||
}
|
|
||||||
|
|
||||||
function shouldExclude(relative: string, exclude: readonly string[]): boolean {
|
|
||||||
return exclude.some((candidate) => isRelativePathOrDescendant(relative, candidate));
|
|
||||||
}
|
|
||||||
|
|
||||||
async function hashFile(filePath: string): Promise<string> {
|
async function hashFile(filePath: string): Promise<string> {
|
||||||
return await new Promise((resolve, reject) => {
|
return await new Promise((resolve, reject) => {
|
||||||
const hash = createHash("sha256");
|
const hash = createHash("sha256");
|
||||||
@@ -43,7 +36,7 @@ async function walkDirectory(
|
|||||||
|
|
||||||
for (const entry of entries) {
|
for (const entry of entries) {
|
||||||
const nextRelative = relative ? path.posix.join(relative, entry.name) : entry.name;
|
const nextRelative = relative ? path.posix.join(relative, entry.name) : entry.name;
|
||||||
if (shouldExclude(nextRelative, exclude)) continue;
|
if (shouldExcludePath(nextRelative, exclude)) continue;
|
||||||
|
|
||||||
const fullPath = path.join(root, nextRelative);
|
const fullPath = path.join(root, nextRelative);
|
||||||
const stats = await fs.lstat(fullPath);
|
const stats = await fs.lstat(fullPath);
|
||||||
|
|||||||
@@ -313,6 +313,82 @@ describe("resolveExecutionRunAdapterConfig", () => {
|
|||||||
details: { code: "low_trust_inline_sensitive_env_denied" },
|
details: { code: "low_trust_inline_sensitive_env_denied" },
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("fails push-capability preflight when no GitHub write credential is bound at agent or project scope", async () => {
|
||||||
|
await expect(resolveExecutionRunAdapterConfig({
|
||||||
|
companyId: "company-1",
|
||||||
|
agentId: "agent-1",
|
||||||
|
issueId: "issue-1",
|
||||||
|
executionRunConfig: { env: { AGENT_ONLY: "agent-only" } },
|
||||||
|
projectEnv: { PROJECT_ONLY: "project-only" },
|
||||||
|
requiredScopedEnvBinding: {
|
||||||
|
keys: ["GH_TOKEN", "GITHUB_TOKEN"],
|
||||||
|
consumerScopes: ["agent", "project"],
|
||||||
|
reason: "push_write_credential_missing",
|
||||||
|
remediation: "GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN bound at project or agent scope.",
|
||||||
|
},
|
||||||
|
secretsSvc: {
|
||||||
|
resolveAdapterConfigForRuntime: vi.fn(),
|
||||||
|
resolveEnvBindings: vi.fn(),
|
||||||
|
} as any,
|
||||||
|
})).rejects.toMatchObject({
|
||||||
|
code: "configuration_incomplete",
|
||||||
|
message: expect.stringContaining("GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN"),
|
||||||
|
resultJson: {
|
||||||
|
configurationIncomplete: {
|
||||||
|
reason: "push_write_credential_missing",
|
||||||
|
requiredEnvKeys: ["GH_TOKEN", "GITHUB_TOKEN"],
|
||||||
|
requiredScopes: ["agent", "project"],
|
||||||
|
missingBindings: [],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("passes push-capability preflight when a project-scoped GitHub credential is configured", async () => {
|
||||||
|
const resolveAdapterConfigForRuntime = vi.fn().mockResolvedValue({
|
||||||
|
config: { env: { AGENT_ONLY: "agent-only" } },
|
||||||
|
secretKeys: new Set<string>(),
|
||||||
|
manifest: [],
|
||||||
|
});
|
||||||
|
const resolveEnvBindings = vi.fn().mockResolvedValue({
|
||||||
|
env: { GH_TOKEN: "github-token" },
|
||||||
|
secretKeys: new Set(["GH_TOKEN"]),
|
||||||
|
manifest: [],
|
||||||
|
});
|
||||||
|
const collectMissingRuntimeBindings = vi.fn().mockResolvedValue([]);
|
||||||
|
|
||||||
|
const result = await resolveExecutionRunAdapterConfig({
|
||||||
|
companyId: "company-1",
|
||||||
|
agentId: "agent-1",
|
||||||
|
issueId: "issue-1",
|
||||||
|
projectId: "project-1",
|
||||||
|
executionRunConfig: { env: { AGENT_ONLY: "agent-only" } },
|
||||||
|
projectEnv: { GH_TOKEN: { type: "plain", value: "github-token" } },
|
||||||
|
requiredScopedEnvBinding: {
|
||||||
|
keys: ["GH_TOKEN", "GITHUB_TOKEN"],
|
||||||
|
consumerScopes: ["agent", "project"],
|
||||||
|
reason: "push_write_credential_missing",
|
||||||
|
remediation: "GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN bound at project or agent scope.",
|
||||||
|
},
|
||||||
|
secretsSvc: {
|
||||||
|
resolveAdapterConfigForRuntime,
|
||||||
|
resolveEnvBindings,
|
||||||
|
collectMissingRuntimeBindings,
|
||||||
|
} as any,
|
||||||
|
});
|
||||||
|
|
||||||
|
expect(result.resolvedConfig.env).toEqual({
|
||||||
|
AGENT_ONLY: "agent-only",
|
||||||
|
GH_TOKEN: "github-token",
|
||||||
|
});
|
||||||
|
expect(resolveEnvBindings).toHaveBeenCalledOnce();
|
||||||
|
expect(collectMissingRuntimeBindings).toHaveBeenCalledTimes(2);
|
||||||
|
expect(collectMissingRuntimeBindings.mock.calls[1]?.[2]).toMatchObject({
|
||||||
|
consumerType: "project",
|
||||||
|
consumerId: "project-1",
|
||||||
|
});
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("extractMentionedSkillIdsFromSources", () => {
|
describe("extractMentionedSkillIdsFromSources", () => {
|
||||||
|
|||||||
@@ -1,3 +1,8 @@
|
|||||||
|
import { execFile as execFileCallback } from "node:child_process";
|
||||||
|
import fs from "node:fs/promises";
|
||||||
|
import os from "node:os";
|
||||||
|
import path from "node:path";
|
||||||
|
import { promisify } from "node:util";
|
||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it } from "vitest";
|
||||||
import type { agents } from "@paperclipai/db";
|
import type { agents } from "@paperclipai/db";
|
||||||
import { sessionCodec as codexSessionCodec } from "@paperclipai/adapter-codex-local/server";
|
import { sessionCodec as codexSessionCodec } from "@paperclipai/adapter-codex-local/server";
|
||||||
@@ -5,6 +10,7 @@ import { resolveDefaultAgentWorkspaceDir } from "../home-paths.js";
|
|||||||
import {
|
import {
|
||||||
applyPersistedExecutionWorkspaceConfig,
|
applyPersistedExecutionWorkspaceConfig,
|
||||||
assertGitSensitiveAdapterWorkspaceValid,
|
assertGitSensitiveAdapterWorkspaceValid,
|
||||||
|
assertPushCapabilityCheckoutValid,
|
||||||
buildRealizedExecutionWorkspaceFromPersisted,
|
buildRealizedExecutionWorkspaceFromPersisted,
|
||||||
buildExplicitResumeSessionOverride,
|
buildExplicitResumeSessionOverride,
|
||||||
deriveTaskKeyWithHeartbeatFallback,
|
deriveTaskKeyWithHeartbeatFallback,
|
||||||
@@ -16,6 +22,7 @@ import {
|
|||||||
prioritizeProjectWorkspaceCandidatesForRun,
|
prioritizeProjectWorkspaceCandidatesForRun,
|
||||||
parseSessionCompactionPolicy,
|
parseSessionCompactionPolicy,
|
||||||
resolveNextSessionState,
|
resolveNextSessionState,
|
||||||
|
requiresPushCapabilityPreflight,
|
||||||
resolveWorkspaceAfterLowTrustPreflight,
|
resolveWorkspaceAfterLowTrustPreflight,
|
||||||
resolveRuntimeSessionParamsForWorkspace,
|
resolveRuntimeSessionParamsForWorkspace,
|
||||||
shouldDeferFollowupWakeForSameIssue,
|
shouldDeferFollowupWakeForSameIssue,
|
||||||
@@ -29,6 +36,8 @@ import {
|
|||||||
} from "../services/heartbeat.ts";
|
} from "../services/heartbeat.ts";
|
||||||
import type { TrustPresetResolution } from "../services/trust-preset-resolver.ts";
|
import type { TrustPresetResolution } from "../services/trust-preset-resolver.ts";
|
||||||
|
|
||||||
|
const execFile = promisify(execFileCallback);
|
||||||
|
|
||||||
function buildResolvedWorkspace(overrides: Partial<ResolvedWorkspaceForRun> = {}): ResolvedWorkspaceForRun {
|
function buildResolvedWorkspace(overrides: Partial<ResolvedWorkspaceForRun> = {}): ResolvedWorkspaceForRun {
|
||||||
return {
|
return {
|
||||||
cwd: "/tmp/project",
|
cwd: "/tmp/project",
|
||||||
@@ -105,6 +114,19 @@ function buildWorkspaceValidationInput(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function runGit(cwd: string, args: string[]) {
|
||||||
|
await execFile("git", args, { cwd });
|
||||||
|
}
|
||||||
|
|
||||||
|
async function createGitCheckout(options: { withRemote: boolean }) {
|
||||||
|
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-push-preflight-"));
|
||||||
|
await runGit(root, ["init"]);
|
||||||
|
if (options.withRemote) {
|
||||||
|
await runGit(root, ["remote", "add", "origin", "https://github.com/example/repo.git"]);
|
||||||
|
}
|
||||||
|
return root;
|
||||||
|
}
|
||||||
|
|
||||||
async function expectWorkspaceValidationFailure(
|
async function expectWorkspaceValidationFailure(
|
||||||
input: WorkspaceValidationInput,
|
input: WorkspaceValidationInput,
|
||||||
reason: string,
|
reason: string,
|
||||||
@@ -377,6 +399,72 @@ describe("assertGitSensitiveAdapterWorkspaceValid", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("assertPushCapabilityCheckoutValid", () => {
|
||||||
|
it("rejects a GitHub PR workflow checkout without a configured push remote", async () => {
|
||||||
|
const cwd = await createGitCheckout({ withRemote: false });
|
||||||
|
try {
|
||||||
|
await expect(assertPushCapabilityCheckoutValid({
|
||||||
|
enabled: true,
|
||||||
|
issue: {
|
||||||
|
id: "issue-1",
|
||||||
|
identifier: "PAP-1",
|
||||||
|
},
|
||||||
|
cwd,
|
||||||
|
})).rejects.toMatchObject({
|
||||||
|
code: "workspace_validation_failed",
|
||||||
|
message: expect.stringContaining("has no configured push remote"),
|
||||||
|
resultJson: {
|
||||||
|
workspaceValidation: expect.objectContaining({
|
||||||
|
reason: "missing_git_push_remote",
|
||||||
|
issueId: "issue-1",
|
||||||
|
executionWorkspaceCwd: cwd,
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
} finally {
|
||||||
|
await fs.rm(cwd, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it("allows a GitHub PR workflow checkout when a push remote is configured", async () => {
|
||||||
|
const cwd = await createGitCheckout({ withRemote: true });
|
||||||
|
try {
|
||||||
|
await expect(assertPushCapabilityCheckoutValid({
|
||||||
|
enabled: true,
|
||||||
|
issue: {
|
||||||
|
id: "issue-1",
|
||||||
|
identifier: "PAP-1",
|
||||||
|
},
|
||||||
|
cwd,
|
||||||
|
})).resolves.toBeUndefined();
|
||||||
|
} finally {
|
||||||
|
await fs.rm(cwd, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("requiresPushCapabilityPreflight", () => {
|
||||||
|
it("only enables the guard when the issue explicitly mentions the GitHub PR workflow skill", () => {
|
||||||
|
expect(requiresPushCapabilityPreflight({
|
||||||
|
adapterType: "codex_local",
|
||||||
|
issueId: "issue-1",
|
||||||
|
explicitRunScopedSkillKeys: ["paperclipai/bundled/software-development/github-pr-workflow"],
|
||||||
|
})).toBe(true);
|
||||||
|
|
||||||
|
expect(requiresPushCapabilityPreflight({
|
||||||
|
adapterType: "codex_local",
|
||||||
|
issueId: "issue-1",
|
||||||
|
explicitRunScopedSkillKeys: [],
|
||||||
|
})).toBe(false);
|
||||||
|
|
||||||
|
expect(requiresPushCapabilityPreflight({
|
||||||
|
adapterType: "cursor-cloud",
|
||||||
|
issueId: "issue-1",
|
||||||
|
explicitRunScopedSkillKeys: ["paperclipai/bundled/software-development/github-pr-workflow"],
|
||||||
|
})).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("stripHostWorkspaceProvisionForLowTrustSandbox", () => {
|
describe("stripHostWorkspaceProvisionForLowTrustSandbox", () => {
|
||||||
it("removes only the host-side provision command for sandbox-backed low-trust runs", () => {
|
it("removes only the host-side provision command for sandbox-backed low-trust runs", () => {
|
||||||
const config = {
|
const config = {
|
||||||
|
|||||||
@@ -61,12 +61,13 @@ async function waitFor(condition: () => boolean | Promise<boolean>, timeoutMs =
|
|||||||
throw new Error("Timed out waiting for condition");
|
throw new Error("Timed out waiting for condition");
|
||||||
}
|
}
|
||||||
|
|
||||||
async function deleteHeartbeatRunsAfterActivityLogDrains(db: Db) {
|
async function deleteHeartbeatRunsAndWakeupsAfterActivityLogDrains(db: Db) {
|
||||||
let lastError: unknown = null;
|
let lastError: unknown = null;
|
||||||
for (let attempt = 0; attempt < 10; attempt += 1) {
|
for (let attempt = 0; attempt < 10; attempt += 1) {
|
||||||
await db.delete(activityLog);
|
await db.delete(activityLog);
|
||||||
try {
|
try {
|
||||||
await db.delete(heartbeatRuns);
|
await db.delete(heartbeatRuns);
|
||||||
|
await db.delete(agentWakeupRequests);
|
||||||
return;
|
return;
|
||||||
} catch (error) {
|
} catch (error) {
|
||||||
lastError = error;
|
lastError = error;
|
||||||
@@ -527,8 +528,7 @@ describeEmbeddedPostgres("low-trust red-team HTTP route regression suite", () =>
|
|||||||
await db.delete(issueRelations);
|
await db.delete(issueRelations);
|
||||||
await db.delete(activityLog);
|
await db.delete(activityLog);
|
||||||
await db.delete(heartbeatRunEvents);
|
await db.delete(heartbeatRunEvents);
|
||||||
await deleteHeartbeatRunsAfterActivityLogDrains(db);
|
await deleteHeartbeatRunsAndWakeupsAfterActivityLogDrains(db);
|
||||||
await db.delete(agentWakeupRequests);
|
|
||||||
await db.delete(issues);
|
await db.delete(issues);
|
||||||
await db.delete(agentRuntimeState);
|
await db.delete(agentRuntimeState);
|
||||||
await db.delete(agents);
|
await db.delete(agents);
|
||||||
|
|||||||
@@ -260,6 +260,9 @@ const WORKSPACE_VALIDATION_FAILURE_CODE = "workspace_validation_failed";
|
|||||||
const WORKSPACE_VALIDATION_RECOVERY_CAUSE = "workspace_validation_failed";
|
const WORKSPACE_VALIDATION_RECOVERY_CAUSE = "workspace_validation_failed";
|
||||||
const CONFIGURATION_INCOMPLETE_FAILURE_CODE = "configuration_incomplete";
|
const CONFIGURATION_INCOMPLETE_FAILURE_CODE = "configuration_incomplete";
|
||||||
const CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE = "configuration_incomplete";
|
const CONFIGURATION_INCOMPLETE_RECOVERY_CAUSE = "configuration_incomplete";
|
||||||
|
const GITHUB_PR_WORKFLOW_SKILL_KEY = "paperclipai/bundled/software-development/github-pr-workflow";
|
||||||
|
const GITHUB_PR_WORKFLOW_SKILL_SLUG = "github-pr-workflow";
|
||||||
|
const PUSH_CAPABILITY_ENV_KEYS = ["GH_TOKEN", "GITHUB_TOKEN"] as const;
|
||||||
// Keep this in sync with local adapters that require a git workspace before launch.
|
// Keep this in sync with local adapters that require a git workspace before launch.
|
||||||
const GIT_SENSITIVE_LOCAL_ADAPTER_TYPES = new Set([
|
const GIT_SENSITIVE_LOCAL_ADAPTER_TYPES = new Set([
|
||||||
"acpx_local",
|
"acpx_local",
|
||||||
@@ -411,6 +414,34 @@ function formatMissingBindingForOperator(missing: MissingRuntimeBinding): string
|
|||||||
return `secret ${secretLabel} not bound at ${missing.consumerType} ${missing.configPath}`;
|
return `secret ${secretLabel} not bound at ${missing.consumerType} ${missing.configPath}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function isConfiguredEnvBindingValue(binding: unknown) {
|
||||||
|
const parsed = envBindingSchema.safeParse(binding);
|
||||||
|
if (!parsed.success) return false;
|
||||||
|
const value = parsed.data;
|
||||||
|
if (typeof value === "string") return value.trim().length > 0;
|
||||||
|
if (value.type === "plain") return value.value.trim().length > 0;
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
function hasGithubPrWorkflowSkill(desiredSkills: string[]) {
|
||||||
|
return desiredSkills.some((skill) => {
|
||||||
|
const normalized = skill.trim();
|
||||||
|
return normalized === GITHUB_PR_WORKFLOW_SKILL_KEY
|
||||||
|
|| normalized === GITHUB_PR_WORKFLOW_SKILL_SLUG
|
||||||
|
|| normalized.endsWith(`/${GITHUB_PR_WORKFLOW_SKILL_SLUG}`);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export function requiresPushCapabilityPreflight(input: {
|
||||||
|
adapterType: string;
|
||||||
|
issueId: string | null | undefined;
|
||||||
|
explicitRunScopedSkillKeys: string[];
|
||||||
|
}) {
|
||||||
|
return Boolean(input.issueId)
|
||||||
|
&& GIT_SENSITIVE_LOCAL_ADAPTER_TYPES.has(input.adapterType)
|
||||||
|
&& hasGithubPrWorkflowSkill(input.explicitRunScopedSkillKeys);
|
||||||
|
}
|
||||||
|
|
||||||
const LOW_TRUST_SENSITIVE_ENV_KEY_RE =
|
const LOW_TRUST_SENSITIVE_ENV_KEY_RE =
|
||||||
/(api[-_]?key|access[-_]?token|auth(?:_?token)?|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring)/i;
|
/(api[-_]?key|access[-_]?token|auth(?:_?token)?|authorization|bearer|secret|passwd|password|credential|jwt|private[-_]?key|cookie|connectionstring)/i;
|
||||||
|
|
||||||
@@ -466,11 +497,18 @@ export async function resolveExecutionRunAdapterConfig(input: {
|
|||||||
routineEnv?: unknown;
|
routineEnv?: unknown;
|
||||||
secretsSvc: RuntimeConfigSecretResolver;
|
secretsSvc: RuntimeConfigSecretResolver;
|
||||||
trustPreset?: TrustPresetResolution;
|
trustPreset?: TrustPresetResolution;
|
||||||
|
requiredScopedEnvBinding?: {
|
||||||
|
keys: string[];
|
||||||
|
consumerScopes: Array<"agent" | "project">;
|
||||||
|
reason: string;
|
||||||
|
remediation: string;
|
||||||
|
};
|
||||||
}) {
|
}) {
|
||||||
const executionRunConfig = stripPaperclipRuntimeEnvFromAdapterConfig(input.executionRunConfig);
|
const executionRunConfig = stripPaperclipRuntimeEnvFromAdapterConfig(input.executionRunConfig);
|
||||||
const environmentEnv = stripPaperclipRuntimeEnvBindings(input.environmentEnv);
|
const environmentEnv = stripPaperclipRuntimeEnvBindings(input.environmentEnv);
|
||||||
const projectEnv = stripPaperclipRuntimeEnvBindings(input.projectEnv);
|
const projectEnv = stripPaperclipRuntimeEnvBindings(input.projectEnv);
|
||||||
const routineEnv = stripPaperclipRuntimeEnvBindings(input.routineEnv);
|
const routineEnv = stripPaperclipRuntimeEnvBindings(input.routineEnv);
|
||||||
|
const agentEnv = parseObject(executionRunConfig.env);
|
||||||
const lowTrustAllowedBindingIds = input.trustPreset?.kind === "low_trust_review"
|
const lowTrustAllowedBindingIds = input.trustPreset?.kind === "low_trust_review"
|
||||||
? input.trustPreset.boundary.allowedSecretBindingIds ?? []
|
? input.trustPreset.boundary.allowedSecretBindingIds ?? []
|
||||||
: undefined;
|
: undefined;
|
||||||
@@ -480,6 +518,31 @@ export async function resolveExecutionRunAdapterConfig(input: {
|
|||||||
assertLowTrustEnvConfigAllowed(projectEnv, "project.env");
|
assertLowTrustEnvConfigAllowed(projectEnv, "project.env");
|
||||||
assertLowTrustEnvConfigAllowed(routineEnv, "routine.env");
|
assertLowTrustEnvConfigAllowed(routineEnv, "routine.env");
|
||||||
}
|
}
|
||||||
|
const requiredScopedEnvBinding = input.requiredScopedEnvBinding ?? null;
|
||||||
|
const requiredScopedBindingsConfigured = requiredScopedEnvBinding
|
||||||
|
? requiredScopedEnvBinding.keys.some((key) => (
|
||||||
|
requiredScopedEnvBinding.consumerScopes.includes("agent")
|
||||||
|
&& isConfiguredEnvBindingValue(agentEnv[key])
|
||||||
|
) || (
|
||||||
|
requiredScopedEnvBinding.consumerScopes.includes("project")
|
||||||
|
&& isConfiguredEnvBindingValue(projectEnv?.[key])
|
||||||
|
))
|
||||||
|
: false;
|
||||||
|
if (requiredScopedEnvBinding && !requiredScopedBindingsConfigured) {
|
||||||
|
throw new ConfigurationIncompleteFailure(`configuration incomplete: ${requiredScopedEnvBinding.remediation}`, {
|
||||||
|
configurationIncomplete: {
|
||||||
|
reason: requiredScopedEnvBinding.reason,
|
||||||
|
companyId: input.companyId,
|
||||||
|
agentId: input.agentId ?? null,
|
||||||
|
issueId: input.issueId ?? null,
|
||||||
|
projectId: input.projectId ?? null,
|
||||||
|
routineId: input.routineId ?? null,
|
||||||
|
requiredEnvKeys: requiredScopedEnvBinding.keys,
|
||||||
|
requiredScopes: requiredScopedEnvBinding.consumerScopes,
|
||||||
|
missingBindings: [],
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
// Pre-dispatch binding-validation gate: detect declared secret refs that have
|
// Pre-dispatch binding-validation gate: detect declared secret refs that have
|
||||||
// no binding before resolving any secret value. Missing bindings short-circuit
|
// no binding before resolving any secret value. Missing bindings short-circuit
|
||||||
// to a configuration-incomplete blocker routed to a human owner instead of a
|
// to a configuration-incomplete blocker routed to a human owner instead of a
|
||||||
@@ -522,6 +585,33 @@ export async function resolveExecutionRunAdapterConfig(input: {
|
|||||||
)),
|
)),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
if (requiredScopedEnvBinding) {
|
||||||
|
const requiredEnvKeys = new Set(requiredScopedEnvBinding.keys);
|
||||||
|
const requiredScopes = new Set(requiredScopedEnvBinding.consumerScopes);
|
||||||
|
const requiredMissingBindings = missingBindings.filter((binding) =>
|
||||||
|
requiredScopes.has(binding.consumerType as "agent" | "project")
|
||||||
|
&& requiredEnvKeys.has(binding.envKey),
|
||||||
|
);
|
||||||
|
if (requiredMissingBindings.length > 0) {
|
||||||
|
const detail = requiredMissingBindings.map(formatMissingBindingForOperator).join("; ");
|
||||||
|
throw new ConfigurationIncompleteFailure(
|
||||||
|
`configuration incomplete: ${requiredScopedEnvBinding.remediation}; ${detail}`,
|
||||||
|
{
|
||||||
|
configurationIncomplete: {
|
||||||
|
reason: requiredScopedEnvBinding.reason,
|
||||||
|
companyId: input.companyId,
|
||||||
|
agentId: input.agentId ?? null,
|
||||||
|
issueId: input.issueId ?? null,
|
||||||
|
projectId: input.projectId ?? null,
|
||||||
|
routineId: input.routineId ?? null,
|
||||||
|
requiredEnvKeys: requiredScopedEnvBinding.keys,
|
||||||
|
requiredScopes: requiredScopedEnvBinding.consumerScopes,
|
||||||
|
missingBindings: requiredMissingBindings,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
if (missingBindings.length > 0) {
|
if (missingBindings.length > 0) {
|
||||||
const detail = missingBindings.map(formatMissingBindingForOperator).join("; ");
|
const detail = missingBindings.map(formatMissingBindingForOperator).join("; ");
|
||||||
throw new ConfigurationIncompleteFailure(`configuration incomplete: ${detail}`, {
|
throw new ConfigurationIncompleteFailure(`configuration incomplete: ${detail}`, {
|
||||||
@@ -1080,6 +1170,53 @@ function sameResolvedPath(left: string | null | undefined, right: string | null
|
|||||||
return path.resolve(leftPath) === path.resolve(rightPath);
|
return path.resolve(leftPath) === path.resolve(rightPath);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function hasGitPushRemote(cwd: string | null | undefined) {
|
||||||
|
const normalized = readNonEmptyString(cwd);
|
||||||
|
if (!normalized) return false;
|
||||||
|
const remoteNames = await execFile("git", ["remote"], { cwd: normalized })
|
||||||
|
.then((result) =>
|
||||||
|
result.stdout
|
||||||
|
.split(/\r?\n/)
|
||||||
|
.map((value) => value.trim())
|
||||||
|
.filter((value) => value.length > 0),
|
||||||
|
)
|
||||||
|
.catch(() => []);
|
||||||
|
|
||||||
|
for (const remoteName of remoteNames) {
|
||||||
|
const pushUrl = await execFile("git", ["remote", "get-url", "--push", remoteName], { cwd: normalized })
|
||||||
|
.then((result) => readNonEmptyString(result.stdout))
|
||||||
|
.catch(() => null);
|
||||||
|
if (pushUrl) return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function assertPushCapabilityCheckoutValid(input: {
|
||||||
|
enabled: boolean;
|
||||||
|
issue: {
|
||||||
|
id: string;
|
||||||
|
identifier: string | null;
|
||||||
|
} | null;
|
||||||
|
cwd: string | null | undefined;
|
||||||
|
}) {
|
||||||
|
if (!input.enabled || !input.issue) return;
|
||||||
|
const cwd = readNonEmptyString(input.cwd);
|
||||||
|
if (!cwd) return;
|
||||||
|
if (await hasGitPushRemote(cwd)) return;
|
||||||
|
throw new WorkspaceValidationFailure(
|
||||||
|
`Issue ${input.issue.identifier ?? input.issue.id} requested the GitHub PR workflow, but checkout "${cwd}" has no configured push remote. Bind the run to a writable repo checkout before dispatching the agent.`,
|
||||||
|
{
|
||||||
|
workspaceValidation: {
|
||||||
|
reason: "missing_git_push_remote",
|
||||||
|
issueId: input.issue.id,
|
||||||
|
issueIdentifier: input.issue.identifier,
|
||||||
|
executionWorkspaceCwd: cwd,
|
||||||
|
requiredEnvKeys: [...PUSH_CAPABILITY_ENV_KEYS],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
export async function assertGitSensitiveAdapterWorkspaceValid(input: {
|
export async function assertGitSensitiveAdapterWorkspaceValid(input: {
|
||||||
adapterType: string;
|
adapterType: string;
|
||||||
agentId: string;
|
agentId: string;
|
||||||
@@ -8631,6 +8768,16 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||||||
: selectedEnvironmentId
|
: selectedEnvironmentId
|
||||||
? await environmentsSvc.getById(selectedEnvironmentId)
|
? await environmentsSvc.getById(selectedEnvironmentId)
|
||||||
: null;
|
: null;
|
||||||
|
const runScopedMentionedSkillKeys = await resolveRunScopedMentionedSkillKeys({
|
||||||
|
db,
|
||||||
|
companyId: agent.companyId,
|
||||||
|
issueId,
|
||||||
|
});
|
||||||
|
const pushCapabilityPreflightRequired = requiresPushCapabilityPreflight({
|
||||||
|
adapterType: agent.adapterType,
|
||||||
|
issueId,
|
||||||
|
explicitRunScopedSkillKeys: runScopedMentionedSkillKeys,
|
||||||
|
});
|
||||||
const { resolvedConfig, secretKeys, secretManifest } = await resolveExecutionRunAdapterConfig({
|
const { resolvedConfig, secretKeys, secretManifest } = await resolveExecutionRunAdapterConfig({
|
||||||
companyId: agent.companyId,
|
companyId: agent.companyId,
|
||||||
agentId: agent.id,
|
agentId: agent.id,
|
||||||
@@ -8645,6 +8792,15 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||||||
routineEnv: routineEnvContext.env,
|
routineEnv: routineEnvContext.env,
|
||||||
secretsSvc,
|
secretsSvc,
|
||||||
trustPreset,
|
trustPreset,
|
||||||
|
requiredScopedEnvBinding: pushCapabilityPreflightRequired
|
||||||
|
? {
|
||||||
|
keys: [...PUSH_CAPABILITY_ENV_KEYS],
|
||||||
|
consumerScopes: ["agent", "project"],
|
||||||
|
reason: "push_write_credential_missing",
|
||||||
|
remediation:
|
||||||
|
"GitHub PR workflow requires GH_TOKEN or GITHUB_TOKEN bound at project or agent scope.",
|
||||||
|
}
|
||||||
|
: undefined,
|
||||||
});
|
});
|
||||||
if (secretManifest.length > 0) {
|
if (secretManifest.length > 0) {
|
||||||
context.paperclipSecrets = {
|
context.paperclipSecrets = {
|
||||||
@@ -8653,11 +8809,6 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||||||
} else {
|
} else {
|
||||||
delete context.paperclipSecrets;
|
delete context.paperclipSecrets;
|
||||||
}
|
}
|
||||||
const runScopedMentionedSkillKeys = await resolveRunScopedMentionedSkillKeys({
|
|
||||||
db,
|
|
||||||
companyId: agent.companyId,
|
|
||||||
issueId,
|
|
||||||
});
|
|
||||||
const effectiveResolvedConfig = applyRunScopedMentionedSkillKeys(
|
const effectiveResolvedConfig = applyRunScopedMentionedSkillKeys(
|
||||||
resolvedConfig,
|
resolvedConfig,
|
||||||
runScopedMentionedSkillKeys,
|
runScopedMentionedSkillKeys,
|
||||||
@@ -9271,6 +9422,16 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
|||||||
environmentDriver: selectedEnvironment.driver,
|
environmentDriver: selectedEnvironment.driver,
|
||||||
leaseMetadata: activeEnvironmentLease.lease.metadata,
|
leaseMetadata: activeEnvironmentLease.lease.metadata,
|
||||||
});
|
});
|
||||||
|
await assertPushCapabilityCheckoutValid({
|
||||||
|
enabled: pushCapabilityPreflightRequired && executionTarget?.kind === "local",
|
||||||
|
issue: issueRef
|
||||||
|
? {
|
||||||
|
id: issueRef.id,
|
||||||
|
identifier: issueRef.identifier,
|
||||||
|
}
|
||||||
|
: null,
|
||||||
|
cwd: executionWorkspace.cwd,
|
||||||
|
});
|
||||||
const adapterEnv = Object.fromEntries(
|
const adapterEnv = Object.fromEntries(
|
||||||
Object.entries(parseObject(resolvedConfig.env)).filter(
|
Object.entries(parseObject(resolvedConfig.env)).filter(
|
||||||
(entry): entry is [string, string] => typeof entry[0] === "string" && typeof entry[1] === "string",
|
(entry): entry is [string, string] => typeof entry[0] === "string" && typeof entry[1] === "string",
|
||||||
|
|||||||
Reference in New Issue
Block a user