feat(plugin-kubernetes): self-hostable Kubernetes sandbox provider (stage 1/3: plugin package) (#5790)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox providers are the seam that lets agent runs execute in isolated environments; today the only first-party remote provider is Daytona, a hosted third-party service > - Self-hosters running Paperclip on their own infrastructure (often Kubernetes already) have no first-party way to run agent sandboxes on a cluster they control > - That gap matters for teams with data-residency, sovereignty, or cost constraints who cannot or will not send workloads to a hosted sandbox service > - This pull request adds a Kubernetes sandbox-provider plugin as a standalone, workspace-excluded package: it implements every SandboxProvider hook the Daytona provider does, on infrastructure the operator owns > - The benefit is that any Paperclip deployment with a Kubernetes cluster gets multi-tenant, network-isolated, quota-bounded agent sandboxes with zero new external dependencies ## Linked Issues or Issue Description No existing issue. Following the feature template: - **Problem:** Paperclip's remote sandbox execution requires a hosted third-party provider. Self-hosters cannot run agent sandboxes on their own Kubernetes clusters with a first-party provider. - **Proposed solution:** A `@paperclipai/plugin-kubernetes` sandbox-provider plugin with two backends: long-lived sandboxes via the [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) CRD (multi-command exec, adapter-install pattern) and one-shot `batch/v1` Jobs (stable APIs only, no extra controllers). - **Alternatives considered:** Driving kubectl from a generic shell provider (no lifecycle/lease semantics), or requiring a hosted provider (exactly the constraint this removes). ## What Changed This is **stage 1 of 3** of a staged contribution (direction agreed with maintainers): the plugin package alone. Stage 2 (server integration: lease params, provider registration) and stage 3 (agent runtime images + CI) are companion PRs that will be cross-linked from a comment here. - New package `packages/plugins/sandbox-providers/kubernetes` (workspace-excluded, like the path already carved out in `pnpm-workspace.yaml`): src, unit + kind integration tests, operator prerequisite manifests, README, smoke-test guide - Implements the full SandboxProvider hook surface the Daytona provider implements: `validateConfig`, `probe`, `acquireLease`, `resumeLease`, `releaseLease`, `destroyLease`, `realizeWorkspace`, `execute` - Two backends: `sandbox-cr` (default; long-lived pod via the agent-sandbox `Sandbox` CR, supports multi-command exec) and `job` (one-shot `batch/v1` Job; nothing beyond k8s 1.27+ required) - Per-run adapter resolution: one environment serves mixed harnesses; the per-run `adapterType` hint is read through a local optional type extension, so the plugin typechecks and builds against the current plugin SDK and simply falls back to the environment's configured default adapter until stage 2 lands - Exec-env wrapping: the Kubernetes exec API carries no environment, so commands are wrapped to receive the run's env - Fast-upload interception for workspace realization, scoped per lease - Per-tenant isolation: derived namespace per company, RBAC, ResourceQuota, restricted-PSS pod security (runAsNonRoot, drop ALL, seccomp RuntimeDefault, no SA token automount) - Network egress policy in two flavors: native `NetworkPolicy` and `CiliumNetworkPolicy` (FQDN allowlists) - Image allowlist with glob matching, registry override, and per-run image override validation - Per-run Kubernetes Secrets carrying agent credentials, ownerRef'd to the Job or Sandbox CR for cascade GC ## Verification - Standalone build, exactly as the README documents: ```bash cd packages/plugins/sandbox-providers/kubernetes pnpm install --ignore-workspace pnpm test # 147 unit tests, 17 files, all green pnpm typecheck # clean against the in-repo plugin SDK on master pnpm build # dist/ emitted, manifest + worker entrypoints present ``` - A kind-cluster end-to-end integration test is included (`RUN_K8S_INTEGRATION_TESTS=1 pnpm test test/integration/end-to-end-run.test.ts`) - Beyond CI: this provider has been verified in a production multi-tenant deployment against five harnesses (opencode, pi, codex, gemini, claude code) with real billed runs ## Risks - **Zero behavior change for any existing deployment.** The package is workspace-excluded; nothing in the server imports or loads it until stage 2's integration lands. No existing code paths are touched. - The default `sandbox-cr` backend depends on an alpha CRD (`agents.x-k8s.io/v1alpha1`); the README flags this and the `job` backend uses only stable APIs as a fallback. - Risk surface is confined to deployments that explicitly install and configure the plugin. - The default runtime images (`ghcr.io/paperclipai/agent-runtime-*`) are published by the stage 3 companion PR (#7934); until that lands, deployments must point `runtimeImage` at their own images. ## Model Used Claude Opus 4.8 (1M context), extended thinking, with tool use (Claude Code). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots (no UI changes) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending this push) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jannes Stubbemann
committed by
GitHub
GitHub
parent
c139d6c025
commit
05ab45225a
+231
@@ -0,0 +1,231 @@
|
||||
import { describe, it, expect, vi, beforeEach } from "vitest";
|
||||
|
||||
// Mock the kube-client module so the plugin handlers run against injected
|
||||
// fake API clients instead of a real cluster. h.clients is swapped per test.
|
||||
const h = vi.hoisted(() => ({ clients: {} as Record<string, unknown> }));
|
||||
|
||||
vi.mock("../../src/kube-client.js", () => ({
|
||||
createKubeConfig: vi.fn(() => ({})),
|
||||
makeKubeClients: vi.fn(() => h.clients),
|
||||
}));
|
||||
|
||||
import plugin from "../../src/plugin.js";
|
||||
|
||||
const CONFIG = { inCluster: true, backend: "sandbox-cr" };
|
||||
|
||||
function leaseMetadata(overrides: Record<string, unknown> = {}): Record<string, unknown> {
|
||||
return {
|
||||
namespace: "paperclip-acme",
|
||||
jobName: "pc-abc",
|
||||
podName: "pc-abc-pod",
|
||||
secretName: "pc-abc-env",
|
||||
phase: "Pending",
|
||||
backend: "sandbox-cr",
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function notFound(): Error {
|
||||
return Object.assign(new Error("not found"), { code: 404 });
|
||||
}
|
||||
|
||||
function readySandboxCr(podName: string): Record<string, unknown> {
|
||||
return {
|
||||
metadata: { uid: "uid-1" },
|
||||
status: {
|
||||
conditions: [{ type: "Ready", status: "True" }],
|
||||
podName,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
beforeEach(() => {
|
||||
h.clients = {};
|
||||
});
|
||||
|
||||
describe("onEnvironmentResumeLease", () => {
|
||||
it("is implemented (Daytona feature parity)", () => {
|
||||
expect(plugin.definition.onEnvironmentResumeLease).toBeTypeOf("function");
|
||||
expect(plugin.definition.onEnvironmentDestroyLease).toBeTypeOf("function");
|
||||
});
|
||||
|
||||
it("returns a valid lease handle for a live sandbox-cr lease", async () => {
|
||||
h.clients = {
|
||||
custom: {
|
||||
getNamespacedCustomObject: vi.fn().mockResolvedValue(readySandboxCr("pc-abc-pod")),
|
||||
},
|
||||
core: {
|
||||
readNamespacedPod: vi.fn().mockResolvedValue({
|
||||
metadata: {},
|
||||
status: { phase: "Running" },
|
||||
}),
|
||||
},
|
||||
};
|
||||
|
||||
const lease = await plugin.definition.onEnvironmentResumeLease!({
|
||||
driverKey: "kubernetes",
|
||||
companyId: "acme",
|
||||
environmentId: "env-1",
|
||||
config: CONFIG,
|
||||
providerLeaseId: "pc-abc",
|
||||
leaseMetadata: leaseMetadata(),
|
||||
});
|
||||
|
||||
expect(lease.providerLeaseId).toBe("pc-abc");
|
||||
expect(lease.metadata).toEqual(
|
||||
expect.objectContaining({
|
||||
namespace: "paperclip-acme",
|
||||
jobName: "pc-abc",
|
||||
podName: "pc-abc-pod",
|
||||
secretName: "pc-abc-env",
|
||||
phase: "Running",
|
||||
backend: "sandbox-cr",
|
||||
resumedLease: true,
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("returns providerLeaseId null (expired) when the Sandbox CR is gone, so the caller falls back to acquireLease", async () => {
|
||||
h.clients = {
|
||||
custom: { getNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()) },
|
||||
core: { readNamespacedPod: vi.fn() },
|
||||
};
|
||||
|
||||
const lease = await plugin.definition.onEnvironmentResumeLease!({
|
||||
driverKey: "kubernetes",
|
||||
companyId: "acme",
|
||||
environmentId: "env-1",
|
||||
config: CONFIG,
|
||||
providerLeaseId: "pc-abc",
|
||||
leaseMetadata: leaseMetadata(),
|
||||
});
|
||||
|
||||
expect(lease.providerLeaseId).toBeNull();
|
||||
expect(lease.metadata?.expired).toBe(true);
|
||||
expect(lease.metadata?.reason).toMatch(/no longer exists/);
|
||||
});
|
||||
|
||||
it("returns providerLeaseId null when the backing pod is gone", async () => {
|
||||
h.clients = {
|
||||
custom: {
|
||||
getNamespacedCustomObject: vi.fn().mockResolvedValue(readySandboxCr("pc-abc-pod")),
|
||||
},
|
||||
core: { readNamespacedPod: vi.fn().mockRejectedValue(notFound()) },
|
||||
};
|
||||
|
||||
const lease = await plugin.definition.onEnvironmentResumeLease!({
|
||||
driverKey: "kubernetes",
|
||||
companyId: "acme",
|
||||
environmentId: "env-1",
|
||||
config: CONFIG,
|
||||
providerLeaseId: "pc-abc",
|
||||
leaseMetadata: leaseMetadata(),
|
||||
});
|
||||
|
||||
expect(lease.providerLeaseId).toBeNull();
|
||||
expect(lease.metadata?.expired).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("onEnvironmentDestroyLease", () => {
|
||||
it("deletes the Sandbox CR, pod, and per-run Secret", async () => {
|
||||
const deleteCr = vi.fn().mockResolvedValue({});
|
||||
const deletePod = vi.fn().mockResolvedValue({});
|
||||
const deleteSecret = vi.fn().mockResolvedValue({});
|
||||
h.clients = {
|
||||
custom: { deleteNamespacedCustomObject: deleteCr },
|
||||
core: { deleteNamespacedPod: deletePod, deleteNamespacedSecret: deleteSecret },
|
||||
batch: { deleteNamespacedJob: vi.fn() },
|
||||
};
|
||||
|
||||
await plugin.definition.onEnvironmentDestroyLease!({
|
||||
driverKey: "kubernetes",
|
||||
companyId: "acme",
|
||||
environmentId: "env-1",
|
||||
config: CONFIG,
|
||||
providerLeaseId: "pc-abc",
|
||||
leaseMetadata: leaseMetadata(),
|
||||
});
|
||||
|
||||
expect(deleteCr).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ namespace: "paperclip-acme", name: "pc-abc" }),
|
||||
);
|
||||
expect(deletePod).toHaveBeenCalledWith({
|
||||
namespace: "paperclip-acme",
|
||||
name: "pc-abc-pod",
|
||||
});
|
||||
expect(deleteSecret).toHaveBeenCalledWith({
|
||||
namespace: "paperclip-acme",
|
||||
name: "pc-abc-env",
|
||||
});
|
||||
});
|
||||
|
||||
it("is idempotent: resolves cleanly when every resource is already gone (404)", async () => {
|
||||
h.clients = {
|
||||
custom: { deleteNamespacedCustomObject: vi.fn().mockRejectedValue(notFound()) },
|
||||
core: {
|
||||
deleteNamespacedPod: vi.fn().mockRejectedValue(notFound()),
|
||||
deleteNamespacedSecret: vi.fn().mockRejectedValue(notFound()),
|
||||
},
|
||||
batch: { deleteNamespacedJob: vi.fn() },
|
||||
};
|
||||
|
||||
await expect(
|
||||
plugin.definition.onEnvironmentDestroyLease!({
|
||||
driverKey: "kubernetes",
|
||||
companyId: "acme",
|
||||
environmentId: "env-1",
|
||||
config: CONFIG,
|
||||
providerLeaseId: "pc-abc",
|
||||
leaseMetadata: leaseMetadata(),
|
||||
}),
|
||||
).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it("is a no-op when providerLeaseId is null", async () => {
|
||||
const deleteCr = vi.fn();
|
||||
h.clients = {
|
||||
custom: { deleteNamespacedCustomObject: deleteCr },
|
||||
core: { deleteNamespacedPod: vi.fn(), deleteNamespacedSecret: vi.fn() },
|
||||
batch: { deleteNamespacedJob: vi.fn() },
|
||||
};
|
||||
|
||||
await plugin.definition.onEnvironmentDestroyLease!({
|
||||
driverKey: "kubernetes",
|
||||
companyId: "acme",
|
||||
environmentId: "env-1",
|
||||
config: CONFIG,
|
||||
providerLeaseId: null,
|
||||
leaseMetadata: undefined,
|
||||
});
|
||||
|
||||
expect(deleteCr).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("deletes the Job for job-backend leases", async () => {
|
||||
const deleteJob = vi.fn().mockResolvedValue({});
|
||||
const deleteCr = vi.fn();
|
||||
h.clients = {
|
||||
custom: { deleteNamespacedCustomObject: deleteCr },
|
||||
core: {
|
||||
deleteNamespacedPod: vi.fn().mockResolvedValue({}),
|
||||
deleteNamespacedSecret: vi.fn().mockResolvedValue({}),
|
||||
},
|
||||
batch: { deleteNamespacedJob: deleteJob },
|
||||
};
|
||||
|
||||
await plugin.definition.onEnvironmentDestroyLease!({
|
||||
driverKey: "kubernetes",
|
||||
companyId: "acme",
|
||||
environmentId: "env-1",
|
||||
config: { inCluster: true, backend: "job" },
|
||||
providerLeaseId: "pc-job",
|
||||
leaseMetadata: leaseMetadata({ jobName: "pc-job", backend: "job", podName: "pc-job-pod", secretName: "pc-job-env" }),
|
||||
});
|
||||
|
||||
expect(deleteJob).toHaveBeenCalledWith(
|
||||
expect.objectContaining({ namespace: "paperclip-acme", name: "pc-job" }),
|
||||
);
|
||||
expect(deleteCr).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user