05ab45225a
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox providers are the seam that lets agent runs execute in isolated environments; today the only first-party remote provider is Daytona, a hosted third-party service > - Self-hosters running Paperclip on their own infrastructure (often Kubernetes already) have no first-party way to run agent sandboxes on a cluster they control > - That gap matters for teams with data-residency, sovereignty, or cost constraints who cannot or will not send workloads to a hosted sandbox service > - This pull request adds a Kubernetes sandbox-provider plugin as a standalone, workspace-excluded package: it implements every SandboxProvider hook the Daytona provider does, on infrastructure the operator owns > - The benefit is that any Paperclip deployment with a Kubernetes cluster gets multi-tenant, network-isolated, quota-bounded agent sandboxes with zero new external dependencies ## Linked Issues or Issue Description No existing issue. Following the feature template: - **Problem:** Paperclip's remote sandbox execution requires a hosted third-party provider. Self-hosters cannot run agent sandboxes on their own Kubernetes clusters with a first-party provider. - **Proposed solution:** A `@paperclipai/plugin-kubernetes` sandbox-provider plugin with two backends: long-lived sandboxes via the [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) CRD (multi-command exec, adapter-install pattern) and one-shot `batch/v1` Jobs (stable APIs only, no extra controllers). - **Alternatives considered:** Driving kubectl from a generic shell provider (no lifecycle/lease semantics), or requiring a hosted provider (exactly the constraint this removes). ## What Changed This is **stage 1 of 3** of a staged contribution (direction agreed with maintainers): the plugin package alone. Stage 2 (server integration: lease params, provider registration) and stage 3 (agent runtime images + CI) are companion PRs that will be cross-linked from a comment here. - New package `packages/plugins/sandbox-providers/kubernetes` (workspace-excluded, like the path already carved out in `pnpm-workspace.yaml`): src, unit + kind integration tests, operator prerequisite manifests, README, smoke-test guide - Implements the full SandboxProvider hook surface the Daytona provider implements: `validateConfig`, `probe`, `acquireLease`, `resumeLease`, `releaseLease`, `destroyLease`, `realizeWorkspace`, `execute` - Two backends: `sandbox-cr` (default; long-lived pod via the agent-sandbox `Sandbox` CR, supports multi-command exec) and `job` (one-shot `batch/v1` Job; nothing beyond k8s 1.27+ required) - Per-run adapter resolution: one environment serves mixed harnesses; the per-run `adapterType` hint is read through a local optional type extension, so the plugin typechecks and builds against the current plugin SDK and simply falls back to the environment's configured default adapter until stage 2 lands - Exec-env wrapping: the Kubernetes exec API carries no environment, so commands are wrapped to receive the run's env - Fast-upload interception for workspace realization, scoped per lease - Per-tenant isolation: derived namespace per company, RBAC, ResourceQuota, restricted-PSS pod security (runAsNonRoot, drop ALL, seccomp RuntimeDefault, no SA token automount) - Network egress policy in two flavors: native `NetworkPolicy` and `CiliumNetworkPolicy` (FQDN allowlists) - Image allowlist with glob matching, registry override, and per-run image override validation - Per-run Kubernetes Secrets carrying agent credentials, ownerRef'd to the Job or Sandbox CR for cascade GC ## Verification - Standalone build, exactly as the README documents: ```bash cd packages/plugins/sandbox-providers/kubernetes pnpm install --ignore-workspace pnpm test # 147 unit tests, 17 files, all green pnpm typecheck # clean against the in-repo plugin SDK on master pnpm build # dist/ emitted, manifest + worker entrypoints present ``` - A kind-cluster end-to-end integration test is included (`RUN_K8S_INTEGRATION_TESTS=1 pnpm test test/integration/end-to-end-run.test.ts`) - Beyond CI: this provider has been verified in a production multi-tenant deployment against five harnesses (opencode, pi, codex, gemini, claude code) with real billed runs ## Risks - **Zero behavior change for any existing deployment.** The package is workspace-excluded; nothing in the server imports or loads it until stage 2's integration lands. No existing code paths are touched. - The default `sandbox-cr` backend depends on an alpha CRD (`agents.x-k8s.io/v1alpha1`); the README flags this and the `job` backend uses only stable APIs as a fallback. - Risk surface is confined to deployments that explicitly install and configure the plugin. - The default runtime images (`ghcr.io/paperclipai/agent-runtime-*`) are published by the stage 3 companion PR (#7934); until that lands, deployments must point `runtimeImage` at their own images. ## Model Used Claude Opus 4.8 (1M context), extended thinking, with tool use (Claude Code). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots (no UI changes) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending this push) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
142 lines
5.1 KiB
TypeScript
142 lines
5.1 KiB
TypeScript
/**
|
|
* Builds a kubernetes-sigs/agent-sandbox Sandbox CR manifest.
|
|
*
|
|
* The Sandbox CR creates a long-lived pod (sleep infinity entrypoint) into
|
|
* which paperclip-server can exec arbitrary commands. This solves the
|
|
* architectural mismatch with the batch/v1 Job backend, which only supports
|
|
* a single one-shot entrypoint — not the multi-command adapter-install pattern
|
|
* used by paperclip-server.
|
|
*
|
|
* Security baseline is identical to buildJobManifest (pod-spec-builder.ts):
|
|
* non-root, drop ALL caps, read-only rootFS, Tini PID 1, seccomp
|
|
* RuntimeDefault, fsGroupChangePolicy OnRootMismatch, automountSAToken=false.
|
|
*
|
|
* NOTE: paperclip-server runs OUTSIDE the cluster, so we cannot set ownerReferences
|
|
* on the Sandbox CR (the owner would need to be an in-cluster resource). The
|
|
* release path is explicit delete via sandboxCrOrchestrator.release().
|
|
*/
|
|
|
|
export interface BuildSandboxCrManifestInput {
|
|
namespace: string;
|
|
sandboxName: string;
|
|
adapterType: string;
|
|
image: string;
|
|
envSecretName: string;
|
|
serviceAccountName: string;
|
|
labels: Record<string, string>;
|
|
resources: {
|
|
requests?: { cpu?: string; memory?: string };
|
|
limits?: { cpu?: string; memory?: string };
|
|
};
|
|
runtimeClassName?: string;
|
|
imagePullSecrets?: string[];
|
|
}
|
|
|
|
export function buildSandboxCrManifest(
|
|
input: BuildSandboxCrManifestInput,
|
|
): Record<string, unknown> {
|
|
const podLabels: Record<string, string> = {
|
|
...input.labels,
|
|
"paperclip.io/role": "agent",
|
|
};
|
|
return {
|
|
apiVersion: "agents.x-k8s.io/v1alpha1",
|
|
kind: "Sandbox",
|
|
metadata: {
|
|
name: input.sandboxName,
|
|
namespace: input.namespace,
|
|
labels: { ...input.labels },
|
|
// No ownerReferences: paperclip-server is out-of-cluster. Release is
|
|
// explicit delete.
|
|
},
|
|
spec: {
|
|
podTemplate: {
|
|
metadata: {
|
|
labels: podLabels,
|
|
},
|
|
spec: {
|
|
serviceAccountName: input.serviceAccountName,
|
|
// Agent containers call back to paperclip-server via HTTPS egress;
|
|
// they never call the Kubernetes API, so mounting an SA token is
|
|
// unnecessary attack surface.
|
|
automountServiceAccountToken: false,
|
|
// Sandbox controller requires restartPolicy: Always so the pod
|
|
// stays running between exec calls.
|
|
restartPolicy: "Always",
|
|
...(input.runtimeClassName
|
|
? { runtimeClassName: input.runtimeClassName }
|
|
: {}),
|
|
...(input.imagePullSecrets && input.imagePullSecrets.length > 0
|
|
? {
|
|
imagePullSecrets: input.imagePullSecrets.map((name) => ({
|
|
name,
|
|
})),
|
|
}
|
|
: {}),
|
|
securityContext: {
|
|
runAsNonRoot: true,
|
|
runAsUser: 1000,
|
|
runAsGroup: 1000,
|
|
fsGroup: 1000,
|
|
fsGroupChangePolicy: "OnRootMismatch",
|
|
seccompProfile: { type: "RuntimeDefault" },
|
|
},
|
|
containers: [
|
|
{
|
|
name: "agent",
|
|
image: input.image,
|
|
imagePullPolicy: "IfNotPresent",
|
|
// sleep infinity keeps the pod running; paperclip-server execs
|
|
// commands into it via Kubernetes exec API. Tini as PID 1 for
|
|
// proper signal forwarding and zombie reaping.
|
|
command: [
|
|
"/usr/bin/tini",
|
|
"--",
|
|
"/bin/sh",
|
|
"-c",
|
|
"sleep infinity",
|
|
],
|
|
// HOME must point at a writable mount; the image's default
|
|
// HOME=/home/node is inside the readOnly root filesystem.
|
|
// Claude (and most agent runtimes) silently exit with code 0
|
|
// and no output when HOME is unwritable, so set this explicitly.
|
|
env: [{ name: "HOME", value: "/home/paperclip" }],
|
|
envFrom: [{ secretRef: { name: input.envSecretName } }],
|
|
securityContext: {
|
|
runAsNonRoot: true,
|
|
runAsUser: 1000,
|
|
runAsGroup: 1000,
|
|
readOnlyRootFilesystem: true,
|
|
allowPrivilegeEscalation: false,
|
|
capabilities: { drop: ["ALL"] },
|
|
},
|
|
resources: {
|
|
requests: input.resources.requests ?? {
|
|
cpu: "250m",
|
|
memory: "512Mi",
|
|
},
|
|
limits: input.resources.limits ?? {
|
|
cpu: "2",
|
|
memory: "4Gi",
|
|
},
|
|
},
|
|
volumeMounts: [
|
|
{ name: "workspace", mountPath: "/workspace" },
|
|
{ name: "home", mountPath: "/home/paperclip" },
|
|
{ name: "cache", mountPath: "/home/paperclip/.cache" },
|
|
{ name: "tmp", mountPath: "/tmp" },
|
|
],
|
|
},
|
|
],
|
|
volumes: [
|
|
{ name: "workspace", emptyDir: { sizeLimit: "8Gi" } },
|
|
{ name: "home", emptyDir: { sizeLimit: "1Gi" } },
|
|
{ name: "cache", emptyDir: { sizeLimit: "1Gi" } },
|
|
{ name: "tmp", emptyDir: { sizeLimit: "2Gi" } },
|
|
],
|
|
},
|
|
},
|
|
},
|
|
};
|
|
}
|