05ab45225a
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox providers are the seam that lets agent runs execute in isolated environments; today the only first-party remote provider is Daytona, a hosted third-party service > - Self-hosters running Paperclip on their own infrastructure (often Kubernetes already) have no first-party way to run agent sandboxes on a cluster they control > - That gap matters for teams with data-residency, sovereignty, or cost constraints who cannot or will not send workloads to a hosted sandbox service > - This pull request adds a Kubernetes sandbox-provider plugin as a standalone, workspace-excluded package: it implements every SandboxProvider hook the Daytona provider does, on infrastructure the operator owns > - The benefit is that any Paperclip deployment with a Kubernetes cluster gets multi-tenant, network-isolated, quota-bounded agent sandboxes with zero new external dependencies ## Linked Issues or Issue Description No existing issue. Following the feature template: - **Problem:** Paperclip's remote sandbox execution requires a hosted third-party provider. Self-hosters cannot run agent sandboxes on their own Kubernetes clusters with a first-party provider. - **Proposed solution:** A `@paperclipai/plugin-kubernetes` sandbox-provider plugin with two backends: long-lived sandboxes via the [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) CRD (multi-command exec, adapter-install pattern) and one-shot `batch/v1` Jobs (stable APIs only, no extra controllers). - **Alternatives considered:** Driving kubectl from a generic shell provider (no lifecycle/lease semantics), or requiring a hosted provider (exactly the constraint this removes). ## What Changed This is **stage 1 of 3** of a staged contribution (direction agreed with maintainers): the plugin package alone. Stage 2 (server integration: lease params, provider registration) and stage 3 (agent runtime images + CI) are companion PRs that will be cross-linked from a comment here. - New package `packages/plugins/sandbox-providers/kubernetes` (workspace-excluded, like the path already carved out in `pnpm-workspace.yaml`): src, unit + kind integration tests, operator prerequisite manifests, README, smoke-test guide - Implements the full SandboxProvider hook surface the Daytona provider implements: `validateConfig`, `probe`, `acquireLease`, `resumeLease`, `releaseLease`, `destroyLease`, `realizeWorkspace`, `execute` - Two backends: `sandbox-cr` (default; long-lived pod via the agent-sandbox `Sandbox` CR, supports multi-command exec) and `job` (one-shot `batch/v1` Job; nothing beyond k8s 1.27+ required) - Per-run adapter resolution: one environment serves mixed harnesses; the per-run `adapterType` hint is read through a local optional type extension, so the plugin typechecks and builds against the current plugin SDK and simply falls back to the environment's configured default adapter until stage 2 lands - Exec-env wrapping: the Kubernetes exec API carries no environment, so commands are wrapped to receive the run's env - Fast-upload interception for workspace realization, scoped per lease - Per-tenant isolation: derived namespace per company, RBAC, ResourceQuota, restricted-PSS pod security (runAsNonRoot, drop ALL, seccomp RuntimeDefault, no SA token automount) - Network egress policy in two flavors: native `NetworkPolicy` and `CiliumNetworkPolicy` (FQDN allowlists) - Image allowlist with glob matching, registry override, and per-run image override validation - Per-run Kubernetes Secrets carrying agent credentials, ownerRef'd to the Job or Sandbox CR for cascade GC ## Verification - Standalone build, exactly as the README documents: ```bash cd packages/plugins/sandbox-providers/kubernetes pnpm install --ignore-workspace pnpm test # 147 unit tests, 17 files, all green pnpm typecheck # clean against the in-repo plugin SDK on master pnpm build # dist/ emitted, manifest + worker entrypoints present ``` - A kind-cluster end-to-end integration test is included (`RUN_K8S_INTEGRATION_TESTS=1 pnpm test test/integration/end-to-end-run.test.ts`) - Beyond CI: this provider has been verified in a production multi-tenant deployment against five harnesses (opencode, pi, codex, gemini, claude code) with real billed runs ## Risks - **Zero behavior change for any existing deployment.** The package is workspace-excluded; nothing in the server imports or loads it until stage 2's integration lands. No existing code paths are touched. - The default `sandbox-cr` backend depends on an alpha CRD (`agents.x-k8s.io/v1alpha1`); the README flags this and the `job` backend uses only stable APIs as a fallback. - Risk surface is confined to deployments that explicitly install and configure the plugin. - The default runtime images (`ghcr.io/paperclipai/agent-runtime-*`) are published by the stage 3 companion PR (#7934); until that lands, deployments must point `runtimeImage` at their own images. ## Model Used Claude Opus 4.8 (1M context), extended thinking, with tool use (Claude Code). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots (no UI changes) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending this push) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
126 lines
4.6 KiB
TypeScript
126 lines
4.6 KiB
TypeScript
import type { KubeClients } from "./kube-client.js";
|
|
import type { SandboxOrchestrator, SandboxStatus } from "./sandbox-orchestrator.js";
|
|
|
|
export class JobTimeoutError extends Error {
|
|
constructor(namespace: string, name: string, timeoutMs: number) {
|
|
super(`Job ${namespace}/${name} did not complete within ${timeoutMs}ms`);
|
|
this.name = "JobTimeoutError";
|
|
}
|
|
}
|
|
|
|
export async function createJob(
|
|
clients: KubeClients,
|
|
namespace: string,
|
|
manifest: Record<string, unknown>,
|
|
): Promise<{ uid: string }> {
|
|
const result = await clients.batch.createNamespacedJob({ namespace, body: manifest as never });
|
|
const uid = (result as { metadata?: { uid?: string } }).metadata?.uid;
|
|
if (!uid) throw new Error("Job created without a UID");
|
|
return { uid };
|
|
}
|
|
|
|
export type JobStatus = SandboxStatus;
|
|
|
|
export async function getJobStatus(
|
|
clients: KubeClients,
|
|
namespace: string,
|
|
name: string,
|
|
): Promise<JobStatus> {
|
|
const result = await clients.batch.readNamespacedJobStatus({ namespace, name });
|
|
const body = (result as Record<string, unknown>) ?? {};
|
|
const status = (body.status as Record<string, unknown>) ?? {};
|
|
const active = (status.active as number) ?? 0;
|
|
const succeeded = (status.succeeded as number) ?? 0;
|
|
const failed = (status.failed as number) ?? 0;
|
|
const conditions = (status.conditions as { type: string; status: string; reason?: string; message?: string }[]) ?? [];
|
|
const completed = conditions.find((c) => c.type === "Complete" && c.status === "True");
|
|
const failedCond = conditions.find((c) => c.type === "Failed" && c.status === "True");
|
|
if (failedCond || failed > 0) {
|
|
return { phase: "Failed", complete: false, active, succeeded, failed, reason: failedCond?.reason, message: failedCond?.message };
|
|
}
|
|
if (completed || succeeded > 0) {
|
|
return { phase: "Succeeded", complete: true, active, succeeded, failed };
|
|
}
|
|
if (active > 0) {
|
|
return { phase: "Running", complete: false, active, succeeded, failed };
|
|
}
|
|
return { phase: "Pending", complete: false, active, succeeded, failed };
|
|
}
|
|
|
|
export async function findPodForJob(
|
|
clients: KubeClients,
|
|
namespace: string,
|
|
jobName: string,
|
|
): Promise<string | null> {
|
|
const result = await clients.core.listNamespacedPod({
|
|
namespace,
|
|
labelSelector: `job-name=${jobName}`,
|
|
});
|
|
const items = ((result as { items?: { metadata?: { name?: string }; status?: { phase?: string } }[] }).items) ?? [];
|
|
const running = items.find((p) => p.status?.phase === "Running");
|
|
return (running ?? items[0])?.metadata?.name ?? null;
|
|
}
|
|
|
|
export async function streamPodLogs(
|
|
clients: KubeClients,
|
|
namespace: string,
|
|
podName: string,
|
|
onChunk: (stream: "stdout" | "stderr", text: string) => Promise<void>,
|
|
): Promise<void> {
|
|
// V1 limitation: readNamespacedPodLog returns combined stdout (the kubectl-style
|
|
// log view). stderr is not separately exposed via this API path — agent
|
|
// containers that need stderr/stdout separation should use a sidecar log
|
|
// scraper or wrap their CLI to emit structured output on stdout. We always
|
|
// emit chunks as "stdout"; the "stderr" callback slot in SandboxOrchestrator
|
|
// is unused by the Job-backed implementation.
|
|
const result = await clients.core.readNamespacedPodLog({ namespace, name: podName });
|
|
const text = (result as string) ?? "";
|
|
if (text.length > 0) await onChunk("stdout", text);
|
|
}
|
|
|
|
export async function deleteJob(
|
|
clients: KubeClients,
|
|
namespace: string,
|
|
name: string,
|
|
): Promise<void> {
|
|
await clients.batch.deleteNamespacedJob({
|
|
namespace,
|
|
name,
|
|
propagationPolicy: "Foreground",
|
|
});
|
|
}
|
|
|
|
export async function waitForJobCompletion(
|
|
clients: KubeClients,
|
|
namespace: string,
|
|
name: string,
|
|
opts: { timeoutMs: number; pollMs?: number } = { timeoutMs: 120_000, pollMs: 2000 },
|
|
): Promise<JobStatus> {
|
|
const deadline = Date.now() + opts.timeoutMs;
|
|
const pollMs = opts.pollMs ?? 2000;
|
|
while (Date.now() < deadline) {
|
|
const status = await getJobStatus(clients, namespace, name);
|
|
if (status.phase === "Succeeded" || status.phase === "Failed") return status;
|
|
await sleep(pollMs);
|
|
}
|
|
throw new JobTimeoutError(namespace, name, opts.timeoutMs);
|
|
}
|
|
|
|
function sleep(ms: number): Promise<void> {
|
|
return new Promise((resolve) => setTimeout(resolve, ms));
|
|
}
|
|
|
|
/**
|
|
* Job-backed conformance to SandboxOrchestrator. Plugin.ts imports THIS value
|
|
* (the swap point) — to use a different backend, swap this import for another
|
|
* module exposing a SandboxOrchestrator-shaped default export.
|
|
*/
|
|
export const jobOrchestrator: SandboxOrchestrator = {
|
|
claim: createJob,
|
|
getStatus: getJobStatus,
|
|
findPod: findPodForJob,
|
|
streamLogs: streamPodLogs,
|
|
release: deleteJob,
|
|
waitForCompletion: waitForJobCompletion,
|
|
};
|