Files
paperclip/packages/plugins/sandbox-providers/kubernetes/test/unit/sandbox-cr-orchestrator.test.ts
T
Jannes Stubbemann 05ab45225a feat(plugin-kubernetes): self-hostable Kubernetes sandbox provider (stage 1/3: plugin package) (#5790)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Sandbox providers are the seam that lets agent runs execute in
isolated environments; today the only first-party remote provider is
Daytona, a hosted third-party service
> - Self-hosters running Paperclip on their own infrastructure (often
Kubernetes already) have no first-party way to run agent sandboxes on a
cluster they control
> - That gap matters for teams with data-residency, sovereignty, or cost
constraints who cannot or will not send workloads to a hosted sandbox
service
> - This pull request adds a Kubernetes sandbox-provider plugin as a
standalone, workspace-excluded package: it implements every
SandboxProvider hook the Daytona provider does, on infrastructure the
operator owns
> - The benefit is that any Paperclip deployment with a Kubernetes
cluster gets multi-tenant, network-isolated, quota-bounded agent
sandboxes with zero new external dependencies

## Linked Issues or Issue Description

No existing issue. Following the feature template:

- **Problem:** Paperclip's remote sandbox execution requires a hosted
third-party provider. Self-hosters cannot run agent sandboxes on their
own Kubernetes clusters with a first-party provider.
- **Proposed solution:** A `@paperclipai/plugin-kubernetes`
sandbox-provider plugin with two backends: long-lived sandboxes via the
[kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox)
CRD (multi-command exec, adapter-install pattern) and one-shot
`batch/v1` Jobs (stable APIs only, no extra controllers).
- **Alternatives considered:** Driving kubectl from a generic shell
provider (no lifecycle/lease semantics), or requiring a hosted provider
(exactly the constraint this removes).

## What Changed

This is **stage 1 of 3** of a staged contribution (direction agreed with
maintainers): the plugin package alone. Stage 2 (server integration:
lease params, provider registration) and stage 3 (agent runtime images +
CI) are companion PRs that will be cross-linked from a comment here.

- New package `packages/plugins/sandbox-providers/kubernetes`
(workspace-excluded, like the path already carved out in
`pnpm-workspace.yaml`): src, unit + kind integration tests, operator
prerequisite manifests, README, smoke-test guide
- Implements the full SandboxProvider hook surface the Daytona provider
implements: `validateConfig`, `probe`, `acquireLease`, `resumeLease`,
`releaseLease`, `destroyLease`, `realizeWorkspace`, `execute`
- Two backends: `sandbox-cr` (default; long-lived pod via the
agent-sandbox `Sandbox` CR, supports multi-command exec) and `job`
(one-shot `batch/v1` Job; nothing beyond k8s 1.27+ required)
- Per-run adapter resolution: one environment serves mixed harnesses;
the per-run `adapterType` hint is read through a local optional type
extension, so the plugin typechecks and builds against the current
plugin SDK and simply falls back to the environment's configured default
adapter until stage 2 lands
- Exec-env wrapping: the Kubernetes exec API carries no environment, so
commands are wrapped to receive the run's env
- Fast-upload interception for workspace realization, scoped per lease
- Per-tenant isolation: derived namespace per company, RBAC,
ResourceQuota, restricted-PSS pod security (runAsNonRoot, drop ALL,
seccomp RuntimeDefault, no SA token automount)
- Network egress policy in two flavors: native `NetworkPolicy` and
`CiliumNetworkPolicy` (FQDN allowlists)
- Image allowlist with glob matching, registry override, and per-run
image override validation
- Per-run Kubernetes Secrets carrying agent credentials, ownerRef'd to
the Job or Sandbox CR for cascade GC

## Verification

- Standalone build, exactly as the README documents:
  ```bash
  cd packages/plugins/sandbox-providers/kubernetes
  pnpm install --ignore-workspace
  pnpm test        # 147 unit tests, 17 files, all green
  pnpm typecheck   # clean against the in-repo plugin SDK on master
pnpm build # dist/ emitted, manifest + worker entrypoints present
  ```
- A kind-cluster end-to-end integration test is included
(`RUN_K8S_INTEGRATION_TESTS=1 pnpm test
test/integration/end-to-end-run.test.ts`)
- Beyond CI: this provider has been verified in a production
multi-tenant deployment against five harnesses (opencode, pi, codex,
gemini, claude code) with real billed runs

## Risks

- **Zero behavior change for any existing deployment.** The package is
workspace-excluded; nothing in the server imports or loads it until
stage 2's integration lands. No existing code paths are touched.
- The default `sandbox-cr` backend depends on an alpha CRD
(`agents.x-k8s.io/v1alpha1`); the README flags this and the `job`
backend uses only stable APIs as a fallback.
- Risk surface is confined to deployments that explicitly install and
configure the plugin.
- The default runtime images (`ghcr.io/paperclipai/agent-runtime-*`) are
published by the stage 3 companion PR (#7934); until that lands,
deployments must point `runtimeImage` at their own images.

## Model Used

Claude Opus 4.8 (1M context), extended thinking, with tool use (Claude
Code).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots (no UI changes)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending this push)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-10 21:07:00 -07:00

238 lines
8.4 KiB
TypeScript

import { describe, it, expect, vi } from "vitest";
import {
createSandboxCr,
deleteSandboxCr,
getSandboxCrStatus,
findPodForSandbox,
SandboxCrTimeoutError,
waitForSandboxReady,
} from "../../src/sandbox-cr-orchestrator.js";
const SANDBOX_GROUP = "agents.x-k8s.io";
const SANDBOX_VERSION = "v1alpha1";
const SANDBOX_PLURAL = "sandboxes";
// Helpers to build mock CR objects with given phase
function makeCr(phase: string, podName?: string): Record<string, unknown> {
return {
metadata: { uid: "sandbox-uid-123" },
status: {
phase,
...(podName ? { podName } : {}),
},
};
}
describe("createSandboxCr", () => {
it("calls custom.createNamespacedCustomObject with the correct params", async () => {
const create = vi.fn().mockResolvedValue({ metadata: { uid: "test-uid" } });
const clients = { custom: { createNamespacedCustomObject: create } };
const manifest = {
apiVersion: "agents.x-k8s.io/v1alpha1",
kind: "Sandbox",
metadata: { name: "pc-abc", namespace: "paperclip-acme" },
};
const result = await createSandboxCr(clients as never, "paperclip-acme", manifest);
expect(create).toHaveBeenCalledWith({
group: SANDBOX_GROUP,
version: SANDBOX_VERSION,
namespace: "paperclip-acme",
plural: SANDBOX_PLURAL,
body: manifest,
});
expect(result.uid).toBe("test-uid");
});
it("throws if the API response has no UID", async () => {
const create = vi.fn().mockResolvedValue({ metadata: {} });
const clients = { custom: { createNamespacedCustomObject: create } };
await expect(
createSandboxCr(clients as never, "ns", {}),
).rejects.toThrow("Sandbox CR created without a UID");
});
});
describe("getSandboxCrStatus", () => {
it("maps phase=Ready to SandboxStatus.phase=Running with active=1", async () => {
const get = vi.fn().mockResolvedValue(makeCr("Ready"));
const clients = { custom: { getNamespacedCustomObject: get } };
const status = await getSandboxCrStatus(clients as never, "ns", "pc-abc");
expect(status.phase).toBe("Running");
expect(status.active).toBe(1);
expect(status.complete).toBe(false);
});
it("maps phase=Pending to SandboxStatus.phase=Pending", async () => {
const get = vi.fn().mockResolvedValue(makeCr("Pending"));
const clients = { custom: { getNamespacedCustomObject: get } };
const status = await getSandboxCrStatus(clients as never, "ns", "pc-abc");
expect(status.phase).toBe("Pending");
expect(status.active).toBe(0);
});
it("maps phase=Failed to SandboxStatus.phase=Failed with failed=1", async () => {
const get = vi.fn().mockResolvedValue({
metadata: { uid: "uid-1" },
status: {
phase: "Failed",
conditions: [
{ type: "Failed", reason: "ImagePullFailed", message: "no image" },
],
},
});
const clients = { custom: { getNamespacedCustomObject: get } };
const status = await getSandboxCrStatus(clients as never, "ns", "pc-abc");
expect(status.phase).toBe("Failed");
expect(status.failed).toBe(1);
expect(status.reason).toBe("ImagePullFailed");
});
it("maps phase=Terminating to SandboxStatus.phase=Running with reason=Terminating", async () => {
const get = vi.fn().mockResolvedValue(makeCr("Terminating"));
const clients = { custom: { getNamespacedCustomObject: get } };
const status = await getSandboxCrStatus(clients as never, "ns", "pc-abc");
expect(status.phase).toBe("Running");
expect(status.reason).toBe("Terminating");
});
});
describe("findPodForSandbox", () => {
it("returns status.podName from the Sandbox CR when set", async () => {
const get = vi.fn().mockResolvedValue(makeCr("Ready", "pc-abc-pod-xyz"));
const clients = {
custom: { getNamespacedCustomObject: get },
core: { listNamespacedPod: vi.fn() },
};
const podName = await findPodForSandbox(clients as never, "ns", "pc-abc");
expect(podName).toBe("pc-abc-pod-xyz");
// Should NOT have called listNamespacedPod (primary path succeeded)
expect(clients.core.listNamespacedPod).not.toHaveBeenCalled();
});
it("falls back to pod listing scoped by the unique sandbox-name label", async () => {
const get = vi.fn().mockResolvedValue(makeCr("Pending")); // no podName
const list = vi.fn().mockResolvedValue({
items: [
{
metadata: { name: "pc-abc-001", labels: { "agents.x-k8s.io/sandbox-name": "pc-abc" } },
status: { phase: "Running" },
},
],
});
const clients = {
custom: { getNamespacedCustomObject: get },
core: { listNamespacedPod: list },
};
const podName = await findPodForSandbox(clients as never, "ns", "pc-abc");
expect(list).toHaveBeenCalledWith(
expect.objectContaining({ labelSelector: "agents.x-k8s.io/sandbox-name=pc-abc" }),
);
expect(podName).toBe("pc-abc-001");
});
it("never matches another sandbox's pod by name prefix", async () => {
const get = vi.fn().mockResolvedValue(makeCr("Pending"));
const list = vi.fn().mockResolvedValue({
items: [
{
// Same name prefix, different sandbox label: must NOT match.
metadata: { name: "pc-abc-zzz", labels: { "agents.x-k8s.io/sandbox-name": "pc-abc-zzz" } },
status: { phase: "Running" },
},
],
});
const clients = {
custom: { getNamespacedCustomObject: get },
core: { listNamespacedPod: list },
};
const podName = await findPodForSandbox(clients as never, "ns", "pc-abc");
expect(podName).toBeNull();
});
it("returns null when no pod is found in fallback", async () => {
const get = vi.fn().mockResolvedValue(makeCr("Pending"));
const list = vi.fn().mockResolvedValue({ items: [] });
const clients = {
custom: { getNamespacedCustomObject: get },
core: { listNamespacedPod: list },
};
const podName = await findPodForSandbox(clients as never, "ns", "pc-abc");
expect(podName).toBeNull();
});
});
describe("deleteSandboxCr", () => {
it("calls custom.deleteNamespacedCustomObject with Foreground propagation", async () => {
const del = vi.fn().mockResolvedValue({});
const clients = { custom: { deleteNamespacedCustomObject: del } };
await deleteSandboxCr(clients as never, "ns", "pc-abc");
expect(del).toHaveBeenCalledWith(
expect.objectContaining({
group: SANDBOX_GROUP,
version: SANDBOX_VERSION,
namespace: "ns",
plural: SANDBOX_PLURAL,
name: "pc-abc",
propagationPolicy: "Foreground",
}),
);
});
});
describe("waitForSandboxReady", () => {
it("resolves immediately when Sandbox is already Ready", async () => {
const get = vi.fn().mockResolvedValue(makeCr("Ready"));
const clients = { custom: { getNamespacedCustomObject: get } };
const status = await waitForSandboxReady(
clients as never,
"ns",
"pc-abc",
{ timeoutMs: 5000, pollMs: 10 },
);
expect(status.phase).toBe("Running"); // Ready maps to Running
expect(get).toHaveBeenCalledTimes(1);
});
it("polls until Ready", async () => {
const get = vi
.fn()
.mockResolvedValueOnce(makeCr("Pending"))
.mockResolvedValueOnce(makeCr("Pending"))
.mockResolvedValueOnce(makeCr("Ready"));
const clients = { custom: { getNamespacedCustomObject: get } };
const status = await waitForSandboxReady(
clients as never,
"ns",
"pc-abc",
{ timeoutMs: 5000, pollMs: 10 },
);
expect(status.phase).toBe("Running");
expect(get).toHaveBeenCalledTimes(3);
});
it("throws SandboxCrTimeoutError when deadline is exceeded", async () => {
const get = vi.fn().mockResolvedValue(makeCr("Pending"));
const clients = { custom: { getNamespacedCustomObject: get } };
await expect(
waitForSandboxReady(clients as never, "ns", "pc-abc", {
timeoutMs: 50,
pollMs: 10,
}),
).rejects.toBeInstanceOf(SandboxCrTimeoutError);
});
it("throws an error describing the failure when Sandbox fails", async () => {
const get = vi.fn().mockResolvedValue({
metadata: { uid: "u1" },
status: { phase: "Failed", conditions: [{ type: "Failed", reason: "OOMKilled" }] },
});
const clients = { custom: { getNamespacedCustomObject: get } };
await expect(
waitForSandboxReady(clients as never, "ns", "pc-abc", {
timeoutMs: 5000,
pollMs: 10,
}),
).rejects.toThrow(/failed.*OOMKilled/i);
});
});