feat(plugin-kubernetes): self-hostable Kubernetes sandbox provider (stage 1/3: plugin package) (#5790)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox providers are the seam that lets agent runs execute in isolated environments; today the only first-party remote provider is Daytona, a hosted third-party service > - Self-hosters running Paperclip on their own infrastructure (often Kubernetes already) have no first-party way to run agent sandboxes on a cluster they control > - That gap matters for teams with data-residency, sovereignty, or cost constraints who cannot or will not send workloads to a hosted sandbox service > - This pull request adds a Kubernetes sandbox-provider plugin as a standalone, workspace-excluded package: it implements every SandboxProvider hook the Daytona provider does, on infrastructure the operator owns > - The benefit is that any Paperclip deployment with a Kubernetes cluster gets multi-tenant, network-isolated, quota-bounded agent sandboxes with zero new external dependencies ## Linked Issues or Issue Description No existing issue. Following the feature template: - **Problem:** Paperclip's remote sandbox execution requires a hosted third-party provider. Self-hosters cannot run agent sandboxes on their own Kubernetes clusters with a first-party provider. - **Proposed solution:** A `@paperclipai/plugin-kubernetes` sandbox-provider plugin with two backends: long-lived sandboxes via the [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) CRD (multi-command exec, adapter-install pattern) and one-shot `batch/v1` Jobs (stable APIs only, no extra controllers). - **Alternatives considered:** Driving kubectl from a generic shell provider (no lifecycle/lease semantics), or requiring a hosted provider (exactly the constraint this removes). ## What Changed This is **stage 1 of 3** of a staged contribution (direction agreed with maintainers): the plugin package alone. Stage 2 (server integration: lease params, provider registration) and stage 3 (agent runtime images + CI) are companion PRs that will be cross-linked from a comment here. - New package `packages/plugins/sandbox-providers/kubernetes` (workspace-excluded, like the path already carved out in `pnpm-workspace.yaml`): src, unit + kind integration tests, operator prerequisite manifests, README, smoke-test guide - Implements the full SandboxProvider hook surface the Daytona provider implements: `validateConfig`, `probe`, `acquireLease`, `resumeLease`, `releaseLease`, `destroyLease`, `realizeWorkspace`, `execute` - Two backends: `sandbox-cr` (default; long-lived pod via the agent-sandbox `Sandbox` CR, supports multi-command exec) and `job` (one-shot `batch/v1` Job; nothing beyond k8s 1.27+ required) - Per-run adapter resolution: one environment serves mixed harnesses; the per-run `adapterType` hint is read through a local optional type extension, so the plugin typechecks and builds against the current plugin SDK and simply falls back to the environment's configured default adapter until stage 2 lands - Exec-env wrapping: the Kubernetes exec API carries no environment, so commands are wrapped to receive the run's env - Fast-upload interception for workspace realization, scoped per lease - Per-tenant isolation: derived namespace per company, RBAC, ResourceQuota, restricted-PSS pod security (runAsNonRoot, drop ALL, seccomp RuntimeDefault, no SA token automount) - Network egress policy in two flavors: native `NetworkPolicy` and `CiliumNetworkPolicy` (FQDN allowlists) - Image allowlist with glob matching, registry override, and per-run image override validation - Per-run Kubernetes Secrets carrying agent credentials, ownerRef'd to the Job or Sandbox CR for cascade GC ## Verification - Standalone build, exactly as the README documents: ```bash cd packages/plugins/sandbox-providers/kubernetes pnpm install --ignore-workspace pnpm test # 147 unit tests, 17 files, all green pnpm typecheck # clean against the in-repo plugin SDK on master pnpm build # dist/ emitted, manifest + worker entrypoints present ``` - A kind-cluster end-to-end integration test is included (`RUN_K8S_INTEGRATION_TESTS=1 pnpm test test/integration/end-to-end-run.test.ts`) - Beyond CI: this provider has been verified in a production multi-tenant deployment against five harnesses (opencode, pi, codex, gemini, claude code) with real billed runs ## Risks - **Zero behavior change for any existing deployment.** The package is workspace-excluded; nothing in the server imports or loads it until stage 2's integration lands. No existing code paths are touched. - The default `sandbox-cr` backend depends on an alpha CRD (`agents.x-k8s.io/v1alpha1`); the README flags this and the `job` backend uses only stable APIs as a fallback. - Risk surface is confined to deployments that explicitly install and configure the plugin. - The default runtime images (`ghcr.io/paperclipai/agent-runtime-*`) are published by the stage 3 companion PR (#7934); until that lands, deployments must point `runtimeImage` at their own images. ## Model Used Claude Opus 4.8 (1M context), extended thinking, with tool use (Claude Code). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots (no UI changes) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending this push) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jannes Stubbemann
committed by
GitHub
GitHub
parent
c139d6c025
commit
05ab45225a
@@ -0,0 +1,22 @@
|
||||
import { execSync } from "node:child_process";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
export const KIND_CONTEXT = "kind-paperclip";
|
||||
|
||||
export function readKindKubeconfig(): string {
|
||||
return readFileSync(join(homedir(), ".kube", "config"), "utf-8");
|
||||
}
|
||||
|
||||
export function kubectl(args: string): string {
|
||||
return execSync(`kubectl --context ${KIND_CONTEXT} ${args}`, { encoding: "utf-8" });
|
||||
}
|
||||
|
||||
export function deleteNamespaceIfExists(namespace: string): void {
|
||||
try {
|
||||
kubectl(`delete namespace ${namespace} --wait=true --timeout=60s --ignore-not-found`);
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
}
|
||||
+205
@@ -0,0 +1,205 @@
|
||||
/**
|
||||
* End-to-end integration test against a local kind cluster.
|
||||
*
|
||||
* PREREQUISITES (operator must perform before running this test):
|
||||
* 1. Create the kind cluster:
|
||||
* kind create cluster --name paperclip
|
||||
* 2. Pre-load the alpine image so the Job can start without network access:
|
||||
* docker pull alpine:3.20
|
||||
* docker tag alpine:3.20 localhost/paperclip-agent:latest
|
||||
* kind load docker-image localhost/paperclip-agent:latest --name paperclip
|
||||
* 3. For the sandbox-cr backend test, the agent-sandbox controller must be installed:
|
||||
* kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/install.yaml
|
||||
* And a tini-bearing image pre-loaded (e.g. the same localhost/paperclip-agent:latest
|
||||
* if it includes /usr/bin/tini and /bin/sh).
|
||||
* 4. Set the env var and run:
|
||||
* RUN_K8S_INTEGRATION_TESTS=1 pnpm test
|
||||
*
|
||||
* The namespace is derived from companySlug ("spike-e2e") + namespacePrefix
|
||||
* ("paperclip-"), resolving to "paperclip-spike-e2e".
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from "vitest";
|
||||
import plugin from "../../src/plugin.js";
|
||||
import { createKubeConfig } from "../../src/kube-client.js";
|
||||
import { execInPod } from "../../src/pod-exec.js";
|
||||
import { sandboxCrOrchestrator } from "../../src/sandbox-cr-orchestrator.js";
|
||||
import { deleteNamespaceIfExists, kubectl, readKindKubeconfig } from "./_kind-harness.js";
|
||||
|
||||
const NAMESPACE = "paperclip-spike-e2e";
|
||||
|
||||
describe("plugin-kubernetes end-to-end", () => {
|
||||
beforeAll(() => {
|
||||
if (process.env.RUN_K8S_INTEGRATION_TESTS !== "1") return;
|
||||
deleteNamespaceIfExists(NAMESPACE);
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
if (process.env.RUN_K8S_INTEGRATION_TESTS !== "1") return;
|
||||
deleteNamespaceIfExists(NAMESPACE);
|
||||
});
|
||||
|
||||
// ── Job backend (stable fallback) ─────────────────────────────────────────
|
||||
|
||||
it.runIf(process.env.RUN_K8S_INTEGRATION_TESTS === "1")(
|
||||
"[job backend] acquireLease creates tenant + Job + supporting resources; releaseLease cascade-deletes them",
|
||||
async () => {
|
||||
const kubeconfig = readKindKubeconfig();
|
||||
const config = {
|
||||
inCluster: false,
|
||||
kubeconfig,
|
||||
companySlug: "spike-e2e",
|
||||
adapterType: "claude_local",
|
||||
backend: "job",
|
||||
imageAllowList: [] as string[],
|
||||
podActivityDeadlineSec: 60,
|
||||
jobTtlSecondsAfterFinished: 60,
|
||||
};
|
||||
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease!({
|
||||
driverKey: "kubernetes",
|
||||
config,
|
||||
runId: "r-test-e2e-job",
|
||||
companyId: "11111111-1111-1111-1111-111111111111",
|
||||
environmentId: "env-test",
|
||||
});
|
||||
|
||||
expect(lease.providerLeaseId).toMatch(/^pc-/);
|
||||
|
||||
// Verify the Job exists in the tenant namespace
|
||||
const jobs = kubectl(`get jobs -n ${NAMESPACE} -o name`);
|
||||
expect(jobs).toContain(`job.batch/${lease.providerLeaseId}`);
|
||||
|
||||
// Verify the tenant namespace has the expected supporting resources
|
||||
const all = kubectl(
|
||||
`get sa,role,rolebinding,resourcequota,limitrange,networkpolicy -n ${NAMESPACE} -o name`,
|
||||
);
|
||||
expect(all).toContain("serviceaccount/paperclip-tenant-sa");
|
||||
expect(all).toContain("role.rbac.authorization.k8s.io/paperclip-tenant-role");
|
||||
expect(all).toContain("rolebinding.rbac.authorization.k8s.io/paperclip-tenant-rb");
|
||||
expect(all).toContain("resourcequota/paperclip-quota");
|
||||
expect(all).toContain("limitrange/paperclip-limits");
|
||||
expect(all).toContain("networkpolicy.networking.k8s.io/paperclip-deny-all");
|
||||
expect(all).toContain("networkpolicy.networking.k8s.io/paperclip-egress-allow");
|
||||
|
||||
// Verify the namespace has PSS-restricted labels
|
||||
const ns = kubectl(`get namespace ${NAMESPACE} -o jsonpath='{.metadata.labels}'`);
|
||||
expect(ns).toContain("pod-security.kubernetes.io/enforce");
|
||||
expect(ns).toContain("restricted");
|
||||
|
||||
// Verify the per-run Secret exists (owned by the Job for cascade deletion)
|
||||
const secrets = kubectl(`get secrets -n ${NAMESPACE} -o name`);
|
||||
expect(secrets).toContain(`secret/${lease.providerLeaseId}-env`);
|
||||
|
||||
// Release — deletes the Job with Foreground propagation, which cascade-deletes
|
||||
// the owned Secret via owner references set at acquireLease time.
|
||||
await plugin.definition.onEnvironmentReleaseLease!({
|
||||
driverKey: "kubernetes",
|
||||
config,
|
||||
providerLeaseId: lease.providerLeaseId,
|
||||
leaseMetadata: lease.metadata,
|
||||
companyId: "11111111-1111-1111-1111-111111111111",
|
||||
environmentId: "env-test",
|
||||
});
|
||||
|
||||
// Allow a brief grace window for Foreground propagation to finish.
|
||||
await new Promise((resolve) => setTimeout(resolve, 2000));
|
||||
|
||||
const jobsAfter = kubectl(`get jobs -n ${NAMESPACE} -o name 2>&1 || true`);
|
||||
expect(jobsAfter).not.toContain(`job.batch/${lease.providerLeaseId}`);
|
||||
},
|
||||
180_000,
|
||||
);
|
||||
|
||||
// ── Sandbox-CR backend (alpha, requires agent-sandbox controller) ──────────
|
||||
|
||||
it.runIf(process.env.RUN_K8S_INTEGRATION_TESTS === "1")(
|
||||
"[sandbox-cr backend] acquireLease creates Sandbox CR + supporting resources; pod becomes Ready; execInPod runs echo hello; releaseLease deletes CR",
|
||||
async () => {
|
||||
const kubeconfig = readKindKubeconfig();
|
||||
const config = {
|
||||
inCluster: false,
|
||||
kubeconfig,
|
||||
companySlug: "spike-e2e",
|
||||
adapterType: "claude_local",
|
||||
backend: "sandbox-cr",
|
||||
imageAllowList: [] as string[],
|
||||
podActivityDeadlineSec: 120,
|
||||
jobTtlSecondsAfterFinished: 60,
|
||||
};
|
||||
|
||||
const lease = await plugin.definition.onEnvironmentAcquireLease!({
|
||||
driverKey: "kubernetes",
|
||||
config,
|
||||
runId: "r-test-e2e-sandbox-cr",
|
||||
companyId: "22222222-2222-2222-2222-222222222222",
|
||||
environmentId: "env-test-cr",
|
||||
});
|
||||
|
||||
expect(lease.providerLeaseId).toMatch(/^pc-/);
|
||||
|
||||
// Verify the Sandbox CR exists in the tenant namespace
|
||||
const sandboxes = kubectl(
|
||||
`get sandboxes.agents.x-k8s.io -n ${NAMESPACE} -o name 2>&1`,
|
||||
);
|
||||
expect(sandboxes).toContain(`sandbox.agents.x-k8s.io/${lease.providerLeaseId}`);
|
||||
|
||||
// Verify the per-run Secret exists (owned by the Sandbox CR)
|
||||
const secrets = kubectl(`get secrets -n ${NAMESPACE} -o name`);
|
||||
expect(secrets).toContain(`secret/${lease.providerLeaseId}-env`);
|
||||
|
||||
// Wait for the Sandbox pod to become Ready
|
||||
const kc = createKubeConfig({ inCluster: false, kubeconfig });
|
||||
const { makeKubeClients } = await import("../../src/kube-client.js");
|
||||
const clients = makeKubeClients(kc);
|
||||
|
||||
await sandboxCrOrchestrator.waitForCompletion(
|
||||
clients,
|
||||
NAMESPACE,
|
||||
lease.providerLeaseId,
|
||||
{ timeoutMs: 90_000, pollMs: 3000 },
|
||||
);
|
||||
|
||||
// Resolve the pod name
|
||||
const podName = await sandboxCrOrchestrator.findPod(
|
||||
clients,
|
||||
NAMESPACE,
|
||||
lease.providerLeaseId,
|
||||
);
|
||||
expect(podName).toBeTruthy();
|
||||
|
||||
// Exec a simple echo command into the running pod
|
||||
const execResult = await execInPod(
|
||||
kc,
|
||||
NAMESPACE,
|
||||
podName!,
|
||||
"agent",
|
||||
["echo", "hello"],
|
||||
);
|
||||
|
||||
expect(execResult.exitCode).toBe(0);
|
||||
expect(execResult.stdout.trim()).toBe("hello");
|
||||
|
||||
// Release — deletes the Sandbox CR with Foreground propagation.
|
||||
await plugin.definition.onEnvironmentReleaseLease!({
|
||||
driverKey: "kubernetes",
|
||||
config,
|
||||
providerLeaseId: lease.providerLeaseId,
|
||||
leaseMetadata: lease.metadata,
|
||||
companyId: "22222222-2222-2222-2222-222222222222",
|
||||
environmentId: "env-test-cr",
|
||||
});
|
||||
|
||||
// Allow a brief grace window for Foreground propagation.
|
||||
await new Promise((resolve) => setTimeout(resolve, 3000));
|
||||
|
||||
const sandboxesAfter = kubectl(
|
||||
`get sandboxes.agents.x-k8s.io -n ${NAMESPACE} -o name 2>&1 || true`,
|
||||
);
|
||||
expect(sandboxesAfter).not.toContain(
|
||||
`sandbox.agents.x-k8s.io/${lease.providerLeaseId}`,
|
||||
);
|
||||
},
|
||||
300_000,
|
||||
);
|
||||
});
|
||||
Reference in New Issue
Block a user