From 05ab45225ae77c7b672bff31496467afb7e6628e Mon Sep 17 00:00:00 2001 From: Jannes Stubbemann Date: Thu, 11 Jun 2026 06:07:00 +0200 Subject: [PATCH] feat(plugin-kubernetes): self-hostable Kubernetes sandbox provider (stage 1/3: plugin package) (#5790) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Sandbox providers are the seam that lets agent runs execute in isolated environments; today the only first-party remote provider is Daytona, a hosted third-party service > - Self-hosters running Paperclip on their own infrastructure (often Kubernetes already) have no first-party way to run agent sandboxes on a cluster they control > - That gap matters for teams with data-residency, sovereignty, or cost constraints who cannot or will not send workloads to a hosted sandbox service > - This pull request adds a Kubernetes sandbox-provider plugin as a standalone, workspace-excluded package: it implements every SandboxProvider hook the Daytona provider does, on infrastructure the operator owns > - The benefit is that any Paperclip deployment with a Kubernetes cluster gets multi-tenant, network-isolated, quota-bounded agent sandboxes with zero new external dependencies ## Linked Issues or Issue Description No existing issue. Following the feature template: - **Problem:** Paperclip's remote sandbox execution requires a hosted third-party provider. Self-hosters cannot run agent sandboxes on their own Kubernetes clusters with a first-party provider. - **Proposed solution:** A `@paperclipai/plugin-kubernetes` sandbox-provider plugin with two backends: long-lived sandboxes via the [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) CRD (multi-command exec, adapter-install pattern) and one-shot `batch/v1` Jobs (stable APIs only, no extra controllers). - **Alternatives considered:** Driving kubectl from a generic shell provider (no lifecycle/lease semantics), or requiring a hosted provider (exactly the constraint this removes). ## What Changed This is **stage 1 of 3** of a staged contribution (direction agreed with maintainers): the plugin package alone. Stage 2 (server integration: lease params, provider registration) and stage 3 (agent runtime images + CI) are companion PRs that will be cross-linked from a comment here. - New package `packages/plugins/sandbox-providers/kubernetes` (workspace-excluded, like the path already carved out in `pnpm-workspace.yaml`): src, unit + kind integration tests, operator prerequisite manifests, README, smoke-test guide - Implements the full SandboxProvider hook surface the Daytona provider implements: `validateConfig`, `probe`, `acquireLease`, `resumeLease`, `releaseLease`, `destroyLease`, `realizeWorkspace`, `execute` - Two backends: `sandbox-cr` (default; long-lived pod via the agent-sandbox `Sandbox` CR, supports multi-command exec) and `job` (one-shot `batch/v1` Job; nothing beyond k8s 1.27+ required) - Per-run adapter resolution: one environment serves mixed harnesses; the per-run `adapterType` hint is read through a local optional type extension, so the plugin typechecks and builds against the current plugin SDK and simply falls back to the environment's configured default adapter until stage 2 lands - Exec-env wrapping: the Kubernetes exec API carries no environment, so commands are wrapped to receive the run's env - Fast-upload interception for workspace realization, scoped per lease - Per-tenant isolation: derived namespace per company, RBAC, ResourceQuota, restricted-PSS pod security (runAsNonRoot, drop ALL, seccomp RuntimeDefault, no SA token automount) - Network egress policy in two flavors: native `NetworkPolicy` and `CiliumNetworkPolicy` (FQDN allowlists) - Image allowlist with glob matching, registry override, and per-run image override validation - Per-run Kubernetes Secrets carrying agent credentials, ownerRef'd to the Job or Sandbox CR for cascade GC ## Verification - Standalone build, exactly as the README documents: ```bash cd packages/plugins/sandbox-providers/kubernetes pnpm install --ignore-workspace pnpm test # 147 unit tests, 17 files, all green pnpm typecheck # clean against the in-repo plugin SDK on master pnpm build # dist/ emitted, manifest + worker entrypoints present ``` - A kind-cluster end-to-end integration test is included (`RUN_K8S_INTEGRATION_TESTS=1 pnpm test test/integration/end-to-end-run.test.ts`) - Beyond CI: this provider has been verified in a production multi-tenant deployment against five harnesses (opencode, pi, codex, gemini, claude code) with real billed runs ## Risks - **Zero behavior change for any existing deployment.** The package is workspace-excluded; nothing in the server imports or loads it until stage 2's integration lands. No existing code paths are touched. - The default `sandbox-cr` backend depends on an alpha CRD (`agents.x-k8s.io/v1alpha1`); the README flags this and the `job` backend uses only stable APIs as a fallback. - Risk surface is confined to deployments that explicitly install and configure the plugin. - The default runtime images (`ghcr.io/paperclipai/agent-runtime-*`) are published by the stage 3 companion PR (#7934); until that lands, deployments must point `runtimeImage` at their own images. ## Model Used Claude Opus 4.8 (1M context), extended thinking, with tool use (Claude Code). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] If this change affects the UI, I have included before/after screenshots (no UI changes) - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green (pending this push) - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Claude Opus 4.8 (1M context) --- .../sandbox-providers/kubernetes/.gitignore | 5 + .../sandbox-providers/kubernetes/README.md | 163 ++++ .../sandbox-providers/kubernetes/SMOKE.md | 135 +++ .../manifests/operator-prerequisites.yaml | 22 + .../sandbox-providers/kubernetes/package.json | 60 ++ .../kubernetes/src/adapter-defaults.ts | 129 +++ .../kubernetes/src/adapter-registry.ts | 28 + .../kubernetes/src/cilium-network-policy.ts | 68 ++ .../kubernetes/src/image-allowlist.ts | 59 ++ .../sandbox-providers/kubernetes/src/index.ts | 2 + .../kubernetes/src/job-orchestrator.ts | 125 +++ .../kubernetes/src/kube-client.ts | 44 + .../kubernetes/src/lease-lifecycle.ts | 193 ++++ .../kubernetes/src/manifest.ts | 121 +++ .../kubernetes/src/network-policy.ts | 132 +++ .../kubernetes/src/plugin.ts | 864 ++++++++++++++++++ .../kubernetes/src/pod-exec.ts | 198 ++++ .../kubernetes/src/pod-spec-builder.ts | 99 ++ .../kubernetes/src/sandbox-cr-builder.ts | 141 +++ .../kubernetes/src/sandbox-cr-orchestrator.ts | 295 ++++++ .../kubernetes/src/sandbox-orchestrator.ts | 68 ++ .../kubernetes/src/secret-manager.ts | 52 ++ .../kubernetes/src/tenant-orchestrator.ts | 299 ++++++ .../sandbox-providers/kubernetes/src/types.ts | 93 ++ .../kubernetes/src/upload-interceptor.ts | 146 +++ .../sandbox-providers/kubernetes/src/utils.ts | 61 ++ .../kubernetes/src/worker.ts | 5 + .../test/integration/_kind-harness.ts | 22 + .../test/integration/end-to-end-run.test.ts | 205 +++++ .../test/unit/adapter-defaults.test.ts | 149 +++ .../test/unit/cilium-network-policy.test.ts | 60 ++ .../test/unit/image-allowlist.test.ts | 62 ++ .../test/unit/job-orchestrator.test.ts | 87 ++ .../kubernetes/test/unit/kube-client.test.ts | 47 + .../test/unit/lease-lifecycle.test.ts | 348 +++++++ .../test/unit/network-policy.test.ts | 95 ++ .../test/unit/plugin-lease-lifecycle.test.ts | 231 +++++ .../kubernetes/test/unit/plugin.test.ts | 94 ++ .../test/unit/pod-spec-builder.test.ts | 95 ++ .../test/unit/sandbox-cr-builder.test.ts | 137 +++ .../test/unit/sandbox-cr-orchestrator.test.ts | 237 +++++ .../test/unit/secret-manager.test.ts | 68 ++ .../test/unit/tenant-orchestrator.test.ts | 106 +++ .../kubernetes/test/unit/types.test.ts | 39 + .../kubernetes/test/unit/utils.test.ts | 60 ++ .../test/unit/wrap-command-with-env.test.ts | 44 + .../kubernetes/tsconfig.json | 11 + .../kubernetes/vitest.config.ts | 12 + scripts/release-package-manifest.json | 5 + 49 files changed, 5821 insertions(+) create mode 100644 packages/plugins/sandbox-providers/kubernetes/.gitignore create mode 100644 packages/plugins/sandbox-providers/kubernetes/README.md create mode 100644 packages/plugins/sandbox-providers/kubernetes/SMOKE.md create mode 100644 packages/plugins/sandbox-providers/kubernetes/manifests/operator-prerequisites.yaml create mode 100644 packages/plugins/sandbox-providers/kubernetes/package.json create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/adapter-defaults.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/adapter-registry.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/cilium-network-policy.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/image-allowlist.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/index.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/job-orchestrator.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/kube-client.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/lease-lifecycle.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/manifest.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/network-policy.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/plugin.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/pod-exec.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/pod-spec-builder.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/sandbox-cr-builder.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/sandbox-cr-orchestrator.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/sandbox-orchestrator.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/secret-manager.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/tenant-orchestrator.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/types.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/upload-interceptor.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/utils.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/src/worker.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/integration/_kind-harness.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/integration/end-to-end-run.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/adapter-defaults.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/cilium-network-policy.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/image-allowlist.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/job-orchestrator.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/kube-client.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/lease-lifecycle.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/network-policy.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/plugin-lease-lifecycle.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/plugin.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/pod-spec-builder.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/sandbox-cr-builder.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/sandbox-cr-orchestrator.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/secret-manager.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/tenant-orchestrator.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/types.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/utils.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/test/unit/wrap-command-with-env.test.ts create mode 100644 packages/plugins/sandbox-providers/kubernetes/tsconfig.json create mode 100644 packages/plugins/sandbox-providers/kubernetes/vitest.config.ts diff --git a/packages/plugins/sandbox-providers/kubernetes/.gitignore b/packages/plugins/sandbox-providers/kubernetes/.gitignore new file mode 100644 index 00000000..859e8b0d --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/.gitignore @@ -0,0 +1,5 @@ +node_modules/ +dist/ +# standalone installs generate a local lockfile; the package is built with +# --no-lockfile (workspace-excluded) and must not commit one +pnpm-lock.yaml diff --git a/packages/plugins/sandbox-providers/kubernetes/README.md b/packages/plugins/sandbox-providers/kubernetes/README.md new file mode 100644 index 00000000..88a65925 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/README.md @@ -0,0 +1,163 @@ +# @paperclipai/plugin-kubernetes (alpha) + +First-party Paperclip sandbox-provider plugin for Kubernetes. + +**Alpha:** the default backend (`sandbox-cr`) is built on `kubernetes-sigs/agent-sandbox` v1alpha1 — expect breaking changes as that CRD evolves toward Beta. A stable fallback backend (`job`, using `batch/v1` Job) is available for clusters without agent-sandbox installed, but it does NOT support multi-command exec (paperclip-server's adapter-install pattern requires sandbox-cr). + +## Prerequisites + +### For `sandbox-cr` backend (default, recommended) + +1. A Kubernetes cluster running k8s 1.27+ +2. [`kubernetes-sigs/agent-sandbox`](https://github.com/kubernetes-sigs/agent-sandbox) controller installed in the cluster (alpha — installs the `sandboxes.agents.x-k8s.io/v1alpha1` CRD and controller) +3. Paperclip-server running with access to the cluster (in-cluster via `inCluster: true` or external via `kubeconfig`) + +### For `job` backend (stable fallback) + +1. A Kubernetes cluster running k8s 1.27+ +2. Paperclip-server with cluster access — no additional controllers or CRDs required + +## Installation + +```bash +paperclipai plugin install @paperclipai/plugin-kubernetes +``` + +Or, for local development: + +```bash +paperclipai plugin install --local /path/to/paperclip/packages/plugins/sandbox-providers/kubernetes +``` + +## Backends + +The plugin supports two backend modes, selected via the `backend` config field: + +| Backend | Default | Stability | Multi-command exec | Requires | +|---|---|---|---|---| +| `sandbox-cr` | Yes | Alpha | Yes | `kubernetes-sigs/agent-sandbox` controller | +| `job` | No | Stable | No | Nothing beyond k8s 1.27+ | + +**`sandbox-cr` (default):** Creates a `Sandbox` CR (`agents.x-k8s.io/v1alpha1`) whose controller provisions a long-lived pod running `sleep infinity`. paperclip-server execs individual commands into the running pod — this is the multi-command adapter-install pattern. When you `releaseLease`, the Sandbox CR is deleted and the controller tears down the pod. + +**`job` (stable fallback):** Creates a `batch/v1` Job. The container entrypoint runs once and exits — no multi-command exec possible. Use this when you cannot install agent-sandbox, or when you need strictly stable Kubernetes APIs. Note: paperclip-server's adapter-install pattern will not work in job mode. + +### Migrating from `job` to `sandbox-cr` + +1. Install the agent-sandbox controller: `kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/install.yaml` +2. Update your environment config to set `backend: "sandbox-cr"` (or remove `backend` since `sandbox-cr` is the default) +3. New leases will use the Sandbox CR backend. Existing leases created with `job` mode continue to use job semantics until they are released. + +## Configuration + +Create a `sandbox` environment with `driver: kubernetes`. One of these auth fields is required: + +- `inCluster: true` — use the in-pod ServiceAccount credentials (when paperclip-server runs inside the same cluster). +- `kubeconfig: ` — inline kubeconfig (stored as a company secret). +- `kubeconfigSecretRef: ` — reference to an existing Paperclip secret. + +Common optional fields: + +| Field | Default | Purpose | +|---|---|---| +| `backend` | `"sandbox-cr"` | `sandbox-cr` (alpha, requires agent-sandbox controller) or `job` (stable, one-shot entrypoint). | +| `adapterType` | `"claude_local"` | One of the supported adapter types (claude_local, codex_local, gemini_local, cursor_local, opencode_local, acpx_local, pi_local). Determines runtime image + env keys + egress allow-list. | +| `namespacePrefix` | `"paperclip-"` | Prefix for the per-company tenant namespace. | +| `companySlug` | derived from companyId | Override the auto-derived company slug. | +| `imageRegistry` | (none) | Override the default registry for agent runtime images. | +| `imageAllowList` | `[]` | Glob patterns of allowed `target.imageOverride` values. Empty = no override permitted. | +| `imagePullSecrets` | `[]` | Names of pre-created Docker image pull secrets in the tenant namespace. | +| `egressAllowFqdns` | `[]` | Additional FQDNs (beyond adapter defaults like `api.anthropic.com`). | +| `egressAllowCidrs` | `[]` | Additional CIDRs to allow egress to. | +| `egressMode` | `"standard"` | `standard` (NetworkPolicy + CIDRs) or `cilium` (CiliumNetworkPolicy + FQDN allow-list). | +| `runtimeClassName` | (none) | e.g. `kata-fc` for Firecracker-backed microVMs. Cluster must have the RuntimeClass installed. | +| `serviceAccountAnnotations` | `{}` | Annotations applied to per-tenant ServiceAccount (e.g. IRSA `eks.amazonaws.com/role-arn`). | +| `jobTtlSecondsAfterFinished` | `900` | Seconds after a Job completes before garbage-collection. | +| `podActivityDeadlineSec` | `3600` | Hard ceiling on a single run's wall-clock time. | + +Full JSON Schema in `src/manifest.ts`. + +## What gets created in your cluster + +For each company that runs agents (created lazily on first dispatch): + +``` +Namespace paperclip-{companySlug} (PSS: restricted enforce + audit) +ServiceAccount paperclip-tenant-sa +Role paperclip-tenant-role (only get pods/log) +RoleBinding paperclip-tenant-rb +ResourceQuota paperclip-quota (pods, requests/limits cpu+memory) +LimitRange paperclip-limits (container max/min/default/defaultRequest) +NetworkPolicy paperclip-deny-all (deny ingress + egress baseline) +NetworkPolicy paperclip-egress-allow (DNS + paperclip-server callback + user CIDRs) + OR CiliumNetworkPolicy paperclip-egress-fqdn if egressMode=cilium +``` + +For each agent run (sandbox-cr backend): + +``` +Sandbox CR pc-{ulid} (agents.x-k8s.io/v1alpha1; explicit delete on release) +Pod pc-{ulid}-{podSuffix} (managed by Sandbox controller; torn down on CR delete) +Secret pc-{ulid}-env (owned by Sandbox CR; cascade-deleted) +``` + +For each agent run (job backend): + +``` +Job pc-{ulid} (backoffLimit: 0, ttlSecondsAfterFinished from config) +Pod pc-{ulid}-{podSuffix} (owned by Job; cascade-deleted) +Secret pc-{ulid}-env (owned by Job; cascade-deleted) +``` + +## Security baseline + +Every agent pod is: + +- non-root (`runAsUser: 1000`, `runAsGroup: 1000`, `runAsNonRoot: true`) +- drops ALL Linux capabilities, `allowPrivilegeEscalation: false` +- `readOnlyRootFilesystem: true` with explicit `emptyDir` mounts for `/workspace`, `/home/paperclip`, `/home/paperclip/.cache`, `/tmp` +- `seccompProfile: RuntimeDefault` +- Tini as PID 1 (reaps zombies, forwards signals) +- `fsGroupChangePolicy: OnRootMismatch` (fast PVC startup; openclaw-operator lesson) +- `automountServiceAccountToken: true` (for the agent shim's paperclip-server callback) + +Plus per-namespace `pod-security.kubernetes.io/enforce: restricted` and a deny-all NetworkPolicy baseline with explicit egress allow-list (DNS, paperclip-server, configured FQDNs/CIDRs). + +The per-run Secret carrying the bootstrap token and adapter API keys has `ownerReferences` pointing at the owning Job, so a single `kubectl delete job …` cascades cleanly to the Pod and Secret. + +## Optional Kata-FC microVM isolation + +For stronger isolation, install [Kata Containers](https://github.com/kata-containers/kata-containers) with the Firecracker hypervisor, then set `runtimeClassName: kata-fc` in the plugin config. Each agent pod will run inside a Firecracker microVM. Requires nested-virt-capable nodes (bare-metal or specific cloud instance types). + +## Roadmap + +- **Phase A (done):** `sandbox-cr` backend — multi-command exec via agent-sandbox Sandbox CRD. +- **Phase B:** Warm pool support — pre-provisioned Sandbox CRs for sub-second cold starts. The `SandboxOrchestrator` interface reserves optional `pause?`/`resume?` extension slots. +- **Phase C:** Kata-FC + snapshots — `runtimeClassName: kata-fc` with VM snapshot for fast restore. +- **Phase D:** Contribute back to agent-sandbox upstream if their Beta model diverges from our needs. The `SandboxOrchestrator` interface (`src/sandbox-orchestrator.ts`) is the clean swap point — a new implementation can be added without touching `plugin.ts` business logic. + +## Lessons learned (from openclaw-operator) + +This plugin adopts patterns from `openclaw-rocks/openclaw-operator`: + +- Tini PID 1 (issue #471 — zombie helper processes) +- Read-only rootFS with explicit writable mounts (issue #456 — ~/.config not writable) +- Strategic merge on reconcile (issue #446 — preserve third-party annotations) +- Multi-storage-class testing (issue #448 — `local-path-provisioner` differences) +- Image version compat matrix (issue #462 — runtime deps cannot resolve after upgrade) + +## Development + +```bash +cd packages/plugins/sandbox-providers/kubernetes +pnpm install --ignore-workspace +pnpm test # unit tests only (fast) +pnpm typecheck +pnpm build +``` + +To run the kind-cluster integration test (requires `kubectl --context kind-paperclip` and a pre-loaded alpine image; see `test/integration/end-to-end-run.test.ts`): + +```bash +RUN_K8S_INTEGRATION_TESTS=1 pnpm test test/integration/end-to-end-run.test.ts +``` diff --git a/packages/plugins/sandbox-providers/kubernetes/SMOKE.md b/packages/plugins/sandbox-providers/kubernetes/SMOKE.md new file mode 100644 index 00000000..9b23828f --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/SMOKE.md @@ -0,0 +1,135 @@ +# Manual smoke test — `@paperclipai/plugin-kubernetes` + +Manual sanity check that the plugin works end-to-end against a real +paperclip-server instance and a real Kubernetes cluster (kind for local +dev). Future work may automate this in CI. + +## Prerequisites + +- A running kind cluster: + ```bash + kind create cluster --name paperclip + ``` +- `kubectl --context kind-paperclip get nodes` returns a node in `Ready` state. + +## Steps + +### 1. Build the plugin + +```bash +cd packages/plugins/sandbox-providers/kubernetes +pnpm install --ignore-workspace +pnpm build +``` + +Expected: `dist/` populated with compiled `.js` and `.d.ts` files. No errors. + +### 2. Start paperclip-server in dev mode + +In a separate terminal: + +```bash +cd /path/to/paperclip +export PAPERCLIP_HOME=/tmp/paperclip-smoke +export PAPERCLIP_INSTANCE_ID=smoke +export PAPERCLIP_DEPLOYMENT_MODE=local_trusted +pnpm --filter @paperclipai/server dev +``` + +Wait for `Server listening on 127.0.0.1:3100`. + +### 3. Install the plugin via the CLI + +```bash +pnpm paperclipai plugin install \ + --local /path/to/paperclip/packages/plugins/sandbox-providers/kubernetes \ + --api-base http://127.0.0.1:3100 +``` + +Expected: `✓ Installed paperclip.kubernetes-sandbox-provider v0.1.0 (ready)`. + +### 4. Create a company and a kubernetes sandbox environment + +```bash +CO_ID=$(curl -s -X POST -H "Content-Type: application/json" \ + -d '{"name":"SmokeCo"}' \ + http://127.0.0.1:3100/api/companies | jq -r '.id') + +KUBECONFIG_CONTENT=$(cat ~/.kube/config | jq -Rs .) + +curl -s -X POST -H "Content-Type: application/json" \ + -d "{ + \"name\": \"k8s-sandbox\", + \"driver\": \"sandbox\", + \"config\": { + \"provider\": \"kubernetes\", + \"kubeconfig\": $KUBECONFIG_CONTENT, + \"companySlug\": \"smoke\", + \"adapterType\": \"claude_local\", + \"imageAllowList\": [\"ghcr.io/paperclipai/agent-runtime-claude:v1\"] + } + }" \ + http://127.0.0.1:3100/api/companies/$CO_ID/environments | jq +``` + +Expected: HTTP 201 with the new environment row. + +### 5. Probe the environment + +```bash +ENV_ID=$(curl -s http://127.0.0.1:3100/api/companies/$CO_ID/environments | jq -r '.[0].id') +curl -s -X POST -d '{}' -H "Content-Type: application/json" \ + http://127.0.0.1:3100/api/environments/$ENV_ID/probe | jq +``` + +Expected: `{"ok": true, ...}` with a summary mentioning the tenant namespace +(`paperclip-smoke`). On first probe the namespace may not yet exist — +the plugin treats a 404 on `listNamespacedPod` as a successful reachability +check. + +### 6. Trigger an agent run + +Use the UI or the API to dispatch a run against the `k8s-sandbox` environment. +The plugin's `onEnvironmentAcquireLease` will: + +1. `ensureTenant` — provision the `paperclip-smoke` namespace, SA, Role, + RoleBinding, ResourceQuota, LimitRange, NetworkPolicies +2. `buildJobManifest` — render the security-hardened Job manifest +3. `createJob` — submit to `batch/v1` +4. `createPerRunSecret` — owned by the Job for cascade-delete + +### 7. Verify the tenant resources + +```bash +kubectl --context kind-paperclip get namespace paperclip-smoke +kubectl --context kind-paperclip get all,networkpolicy,resourcequota,limitrange,sa,role,rolebinding -n paperclip-smoke +``` + +Expected: + +- Namespace `paperclip-smoke` exists with PSS labels + (`pod-security.kubernetes.io/enforce=restricted`) +- ServiceAccount `paperclip-tenant-sa` +- Role `paperclip-tenant-role`, RoleBinding `paperclip-tenant-rb` +- ResourceQuota `paperclip-quota`, LimitRange `paperclip-limits` +- NetworkPolicies `paperclip-deny-all` + `paperclip-egress-allow` +- Job `pc-{ulid}` and its child Pod +- Secret `pc-{ulid}-env` with `ownerReferences` pointing at the Job + +### 8. Tear down + +```bash +kubectl --context kind-paperclip delete namespace paperclip-smoke +kill %1 # paperclip-server +``` + +### 9. Document the result + +In the PR description (or appended to this file as a dated section), +record: + +- Date + git SHA +- `kubectl version` server version +- Output of `kubectl get all -n paperclip-smoke` after step 6 +- Probe response from step 5 +- Time-to-acquire-lease (target: <30s on kind for a cold tenant) diff --git a/packages/plugins/sandbox-providers/kubernetes/manifests/operator-prerequisites.yaml b/packages/plugins/sandbox-providers/kubernetes/manifests/operator-prerequisites.yaml new file mode 100644 index 00000000..1cb9c96a --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/manifests/operator-prerequisites.yaml @@ -0,0 +1,22 @@ +# This plugin uses only stable Kubernetes APIs. No CRD installation is required. +# +# Minimum cluster version: Kubernetes 1.27+ +# - batch/v1 Job (GA since k8s 1.21) +# - core/v1 Pod, Secret, Namespace, ServiceAccount, ResourceQuota, LimitRange (GA since k8s 1.0) +# - rbac.authorization.k8s.io/v1 Role, RoleBinding (GA since k8s 1.8) +# - networking.k8s.io/v1 NetworkPolicy (GA since k8s 1.7) +# - Pod Security Standards namespace labels (GA in k8s 1.25) +# - fsGroupChangePolicy: OnRootMismatch (GA in k8s 1.23) +# - seccompProfile.type: RuntimeDefault (GA in k8s 1.19) +# +# Optional CNI prerequisites for FQDN-based egress (egressMode: cilium): +# - Cilium >= 1.11 with hubble + DNS proxy enabled +# - cilium.io/v2 CiliumNetworkPolicy (provided by Cilium installation) +# +# Optional runtime class for microVM isolation (runtimeClassName: kata-fc): +# - kata-containers with Firecracker hypervisor +# - nested-virt-capable nodes +# +# Future backends (not currently required): +# - kubernetes-sigs/agent-sandbox (when it reaches v1beta1) as an alternative +# backend for warm pools / templates / pause-resume. diff --git a/packages/plugins/sandbox-providers/kubernetes/package.json b/packages/plugins/sandbox-providers/kubernetes/package.json new file mode 100644 index 00000000..4f6fdef6 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/package.json @@ -0,0 +1,60 @@ +{ + "name": "@paperclipai/plugin-kubernetes", + "version": "0.1.0", + "description": "Kubernetes sandbox provider plugin for Paperclip environments", + "license": "MIT", + "homepage": "https://github.com/paperclipai/paperclip", + "bugs": { + "url": "https://github.com/paperclipai/paperclip/issues" + }, + "repository": { + "type": "git", + "url": "https://github.com/paperclipai/paperclip", + "directory": "packages/plugins/sandbox-providers/kubernetes" + }, + "type": "module", + "exports": { + ".": "./src/index.ts" + }, + "publishConfig": { + "access": "public", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + } + }, + "main": "./dist/index.js", + "types": "./dist/index.d.ts" + }, + "files": ["dist", "manifests", "README.md"], + "paperclipPlugin": { + "manifest": "./dist/manifest.js", + "worker": "./dist/worker.js" + }, + "keywords": [ + "paperclip", + "plugin", + "sandbox", + "kubernetes" + ], + "scripts": { + "postinstall": "node ../../../../scripts/link-plugin-dev-sdk.mjs", + "prebuild": "pnpm -C ../../../.. --filter @paperclipai/plugin-sdk ensure-build-deps", + "build": "rm -rf dist && tsc", + "clean": "rm -rf dist", + "typecheck": "pnpm -C ../../../.. --filter @paperclipai/plugin-sdk ensure-build-deps && tsc --noEmit", + "test": "vitest run --config vitest.config.ts", + "prepack": "rm -f package.dev.json && cp package.json package.dev.json && node ../../../../scripts/generate-plugin-package-json.mjs", + "postpack": "if [ -f package.dev.json ]; then mv package.dev.json package.json; fi" + }, + "dependencies": { + "@kubernetes/client-node": "^1.0.0", + "zod": "^3.24.2" + }, + "devDependencies": { + "@types/node": "^24.6.0", + "typescript": "^5.7.3", + "vitest": "^3.2.4" + } +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/adapter-defaults.ts b/packages/plugins/sandbox-providers/kubernetes/src/adapter-defaults.ts new file mode 100644 index 00000000..f30a1216 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/adapter-defaults.ts @@ -0,0 +1,129 @@ +import type { AdapterRegistryEntry } from "./adapter-registry.js"; + +export interface AdapterDefaults { + runtimeImage: string; + envKeys: string[]; + allowFqdns: string[]; + probeCommand: string[]; + /** Non-secret env injected as the base layer for the Job (process-env wins on top). */ + defaultEnv?: Record; +} + +const REGISTRY: Record = { + claude_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-claude:v1", + envKeys: ["ANTHROPIC_API_KEY"], + allowFqdns: ["api.anthropic.com"], + probeCommand: ["claude", "--version"], + }, + codex_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-codex:v1", + envKeys: ["OPENAI_API_KEY"], + allowFqdns: ["api.openai.com"], + probeCommand: ["codex", "--version"], + }, + gemini_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-gemini:v1", + envKeys: ["GOOGLE_API_KEY", "GEMINI_API_KEY"], + allowFqdns: ["generativelanguage.googleapis.com"], + probeCommand: ["gemini", "--version"], + }, + cursor_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-cursor:v1", + envKeys: ["ANTHROPIC_API_KEY", "OPENAI_API_KEY"], + allowFqdns: ["api.anthropic.com", "api.openai.com"], + probeCommand: ["cursor-agent", "--version"], + }, + opencode_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-opencode:v1", + envKeys: ["ANTHROPIC_API_KEY", "OPENAI_API_KEY", "OPENROUTER_API_KEY"], + allowFqdns: ["api.anthropic.com", "api.openai.com", "openrouter.ai"], + probeCommand: ["opencode", "--version"], + }, + acpx_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-acpx:v1", + envKeys: ["ANTHROPIC_API_KEY", "OPENAI_API_KEY"], + allowFqdns: ["api.anthropic.com", "api.openai.com"], + probeCommand: ["acpx", "--version"], + }, + pi_local: { + runtimeImage: "ghcr.io/paperclipai/agent-runtime-pi:v1", + envKeys: ["ANTHROPIC_API_KEY"], + allowFqdns: ["api.anthropic.com"], + probeCommand: ["pi", "--version"], + }, +}; + +export const KNOWN_ADAPTER_TYPES: ReadonlySet = new Set(Object.keys(REGISTRY)); + +function fromRegistryEntry(entry: AdapterRegistryEntry): AdapterDefaults { + // Only runtimeImage is strictly required. The array fields are optional and + // default to []: the operator emits them with `omitempty`, so a genuinely + // empty allowFqdns/envKeys/probeCommand arrives as undefined, which is valid + // (no extra egress / no forwarded secrets / no probe), NOT an error. + if (!entry.runtimeImage) { + throw new Error( + `Adapter "${entry.adapterType}" is missing required runtime field: runtimeImage`, + ); + } + return { + runtimeImage: entry.runtimeImage, + envKeys: entry.envKeys ?? [], + allowFqdns: entry.allowFqdns ?? [], + probeCommand: entry.probeCommand ?? [], + defaultEnv: entry.defaultEnv, + }; +} + +/** + * Resolve the runtime defaults for an adapter. When a `registry` is supplied it + * is authoritative (replace semantics): the type MUST be present and complete, + * else this throws. With no registry, falls back to the built-in REGISTRY. + */ +export function getAdapterDefaults( + adapterType: string, + registry?: readonly AdapterRegistryEntry[], +): AdapterDefaults { + if (registry && registry.length > 0) { + const entry = registry.find((e) => e.adapterType === adapterType); + if (!entry) { + throw new Error(`Adapter "${adapterType}" is not in the configured adapter registry`); + } + return fromRegistryEntry(entry); + } + const defaults = REGISTRY[adapterType]; + if (!defaults) { + throw new Error(`Unknown adapter type: ${adapterType}`); + } + return defaults; +} + +/** + * Resolve the adapter type for a single run: prefer the run's adapter (the agent's, + * from the lease params) so one environment can serve mixed harnesses; fall back to + * the environment's configured default adapter when the run does not specify one. + */ +export function resolveRunAdapterType( + runAdapterType: string | null | undefined, + configAdapterType: string, +): string { + const trimmed = typeof runAdapterType === "string" ? runAdapterType.trim() : ""; + return trimmed.length > 0 ? trimmed : configAdapterType; +} + +/** + * Build the per-run env for the Job: the non-secret `defaultEnv` is the base + * and the process-env values (the secret API keys named by `envKeys`) override + * it. Pure for testability. + */ +export function buildAdapterEnv( + defaults: AdapterDefaults, + processEnv: NodeJS.ProcessEnv = process.env, +): Record { + const out: Record = { ...(defaults.defaultEnv ?? {}) }; + for (const k of defaults.envKeys) { + const v = processEnv[k]; + if (v) out[k] = v; + } + return out; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/adapter-registry.ts b/packages/plugins/sandbox-providers/kubernetes/src/adapter-registry.ts new file mode 100644 index 00000000..ce732b57 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/adapter-registry.ts @@ -0,0 +1,28 @@ +import { z } from "zod"; + +/** + * One declarative agent-harness ("adapter") entry. Governs picker availability + * and, for sandboxed (Kubernetes) runs, the runtime wiring. + * + * NOTE: this shape is intentionally duplicated across the package boundary. It + * MUST stay structurally in sync with: + * - server `@paperclipai/shared` `adapterRegistryEntrySchema` (the parser side) + * - operator `AdapterRegistryEntry` Go struct (PAPERCLIP_ADAPTERS emitter) + * The duplication is deliberate: this plugin is standalone-installable and must + * not pull in heavy workspace packages at runtime. + */ +export const adapterRegistryEntrySchema = z + .object({ + adapterType: z.string().min(1), + enabled: z.boolean().default(true), + runtimeImage: z.string().optional(), + envKeys: z.array(z.string()).optional(), + allowFqdns: z.array(z.string()).optional(), + probeCommand: z.array(z.string()).optional(), + defaultEnv: z.record(z.string()).optional(), + }) + .strict(); + +export const adapterRegistrySchema = z.array(adapterRegistryEntrySchema); + +export type AdapterRegistryEntry = z.infer; diff --git a/packages/plugins/sandbox-providers/kubernetes/src/cilium-network-policy.ts b/packages/plugins/sandbox-providers/kubernetes/src/cilium-network-policy.ts new file mode 100644 index 00000000..5dedcf73 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/cilium-network-policy.ts @@ -0,0 +1,68 @@ +export interface BuildCiliumNetworkPolicyInput { + namespace: string; + paperclipServerNamespace: string; + egressAllowFqdns: string[]; + egressAllowCidrs: string[]; +} + +// Design note: no ingress rules are defined here. Paperclip-server does NOT +// push to agent pods — agents make outbound (egress) callbacks to +// paperclip-server on port 3100. If server→agent push is ever needed, add a +// targeted ingress rule scoped to the paperclip-server endpoint selector. +export function buildCiliumNetworkPolicyManifest(input: BuildCiliumNetworkPolicyInput): Record { + const egress: Record[] = []; + + egress.push({ + toEndpoints: [ + { matchLabels: { "k8s:io.kubernetes.pod.namespace": "kube-system", "k8s-app": "kube-dns" } }, + ], + toPorts: [ + { + ports: [ + { port: "53", protocol: "UDP" }, + { port: "53", protocol: "TCP" }, + ], + rules: { dns: [{ matchPattern: "*" }] }, + }, + ], + }); + + if (input.egressAllowFqdns.length > 0) { + egress.push({ + toFQDNs: input.egressAllowFqdns.map((fqdn) => ({ matchName: fqdn })), + toPorts: [{ ports: [{ port: "443", protocol: "TCP" }] }], + }); + } + + egress.push({ + toEndpoints: [ + { + matchLabels: { + "k8s:io.kubernetes.pod.namespace": input.paperclipServerNamespace, + app: "paperclip-server", + }, + }, + ], + toPorts: [{ ports: [{ port: "3100", protocol: "TCP" }] }], + }); + + if (input.egressAllowCidrs.length > 0) { + egress.push({ + toCIDRSet: input.egressAllowCidrs.map((cidr) => ({ cidr })), + }); + } + + return { + apiVersion: "cilium.io/v2", + kind: "CiliumNetworkPolicy", + metadata: { + name: "paperclip-egress-fqdn", + namespace: input.namespace, + labels: { "paperclip.io/managed-by": "paperclip-k8s-plugin" }, + }, + spec: { + endpointSelector: { matchLabels: { "paperclip.io/role": "agent" } }, + egress, + }, + }; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/image-allowlist.ts b/packages/plugins/sandbox-providers/kubernetes/src/image-allowlist.ts new file mode 100644 index 00000000..46cdaf61 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/image-allowlist.ts @@ -0,0 +1,59 @@ +/** + * Glob matching for image references. + * - `*` matches any sequence of characters EXCEPT `/` (so a wildcard doesn't span path segments) + * - `?` matches exactly one character (excluding `/`) + */ +export function globMatch(pattern: string, value: string): boolean { + const re = new RegExp( + "^" + + pattern + .replace(/[.+^${}()|[\]\\]/g, "\\$&") + .replace(/\*/g, "[^/]*") + .replace(/\?/g, "[^/]") + + "$", + ); + return re.test(value); +} + +export interface ResolveImageInput { + imageOverride?: string | null; +} + +export interface ResolveImageDefaults { + runtimeImage: string; +} + +export interface ResolveImageConfig { + imageAllowList: string[]; + imageRegistry?: string; +} + +export function resolveImage( + target: ResolveImageInput, + defaults: ResolveImageDefaults, + config: ResolveImageConfig, +): string { + if (target.imageOverride) { + if (!config.imageAllowList.some((p) => globMatch(p, target.imageOverride!))) { + throw new Error(`Image override "${target.imageOverride}" is not in allowlist`); + } + return target.imageOverride; + } + if (config.imageRegistry) { + return rewriteRegistry(defaults.runtimeImage, config.imageRegistry); + } + return defaults.runtimeImage; +} + +function rewriteRegistry(image: string, registry: string): string { + // image is like "ghcr.io/paperclipai/agent-runtime-claude:v1" + // we want to replace the first two path segments (host + org) with `registry` + const cleanRegistry = registry.replace(/\/+$/, ""); + const colonIdx = image.lastIndexOf(":"); + const tag = colonIdx >= 0 ? image.slice(colonIdx) : ""; + const path = colonIdx >= 0 ? image.slice(0, colonIdx) : image; + const segments = path.split("/"); + // Strip the host+org (first two segments), keep the image name + const imageName = segments.slice(2).join("/") || segments[segments.length - 1]; + return `${cleanRegistry}/${imageName}${tag}`; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/index.ts b/packages/plugins/sandbox-providers/kubernetes/src/index.ts new file mode 100644 index 00000000..f7ce1cc1 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/index.ts @@ -0,0 +1,2 @@ +export { default as manifest } from "./manifest.js"; +export { default as plugin } from "./plugin.js"; diff --git a/packages/plugins/sandbox-providers/kubernetes/src/job-orchestrator.ts b/packages/plugins/sandbox-providers/kubernetes/src/job-orchestrator.ts new file mode 100644 index 00000000..4260ed0d --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/job-orchestrator.ts @@ -0,0 +1,125 @@ +import type { KubeClients } from "./kube-client.js"; +import type { SandboxOrchestrator, SandboxStatus } from "./sandbox-orchestrator.js"; + +export class JobTimeoutError extends Error { + constructor(namespace: string, name: string, timeoutMs: number) { + super(`Job ${namespace}/${name} did not complete within ${timeoutMs}ms`); + this.name = "JobTimeoutError"; + } +} + +export async function createJob( + clients: KubeClients, + namespace: string, + manifest: Record, +): Promise<{ uid: string }> { + const result = await clients.batch.createNamespacedJob({ namespace, body: manifest as never }); + const uid = (result as { metadata?: { uid?: string } }).metadata?.uid; + if (!uid) throw new Error("Job created without a UID"); + return { uid }; +} + +export type JobStatus = SandboxStatus; + +export async function getJobStatus( + clients: KubeClients, + namespace: string, + name: string, +): Promise { + const result = await clients.batch.readNamespacedJobStatus({ namespace, name }); + const body = (result as Record) ?? {}; + const status = (body.status as Record) ?? {}; + const active = (status.active as number) ?? 0; + const succeeded = (status.succeeded as number) ?? 0; + const failed = (status.failed as number) ?? 0; + const conditions = (status.conditions as { type: string; status: string; reason?: string; message?: string }[]) ?? []; + const completed = conditions.find((c) => c.type === "Complete" && c.status === "True"); + const failedCond = conditions.find((c) => c.type === "Failed" && c.status === "True"); + if (failedCond || failed > 0) { + return { phase: "Failed", complete: false, active, succeeded, failed, reason: failedCond?.reason, message: failedCond?.message }; + } + if (completed || succeeded > 0) { + return { phase: "Succeeded", complete: true, active, succeeded, failed }; + } + if (active > 0) { + return { phase: "Running", complete: false, active, succeeded, failed }; + } + return { phase: "Pending", complete: false, active, succeeded, failed }; +} + +export async function findPodForJob( + clients: KubeClients, + namespace: string, + jobName: string, +): Promise { + const result = await clients.core.listNamespacedPod({ + namespace, + labelSelector: `job-name=${jobName}`, + }); + const items = ((result as { items?: { metadata?: { name?: string }; status?: { phase?: string } }[] }).items) ?? []; + const running = items.find((p) => p.status?.phase === "Running"); + return (running ?? items[0])?.metadata?.name ?? null; +} + +export async function streamPodLogs( + clients: KubeClients, + namespace: string, + podName: string, + onChunk: (stream: "stdout" | "stderr", text: string) => Promise, +): Promise { + // V1 limitation: readNamespacedPodLog returns combined stdout (the kubectl-style + // log view). stderr is not separately exposed via this API path — agent + // containers that need stderr/stdout separation should use a sidecar log + // scraper or wrap their CLI to emit structured output on stdout. We always + // emit chunks as "stdout"; the "stderr" callback slot in SandboxOrchestrator + // is unused by the Job-backed implementation. + const result = await clients.core.readNamespacedPodLog({ namespace, name: podName }); + const text = (result as string) ?? ""; + if (text.length > 0) await onChunk("stdout", text); +} + +export async function deleteJob( + clients: KubeClients, + namespace: string, + name: string, +): Promise { + await clients.batch.deleteNamespacedJob({ + namespace, + name, + propagationPolicy: "Foreground", + }); +} + +export async function waitForJobCompletion( + clients: KubeClients, + namespace: string, + name: string, + opts: { timeoutMs: number; pollMs?: number } = { timeoutMs: 120_000, pollMs: 2000 }, +): Promise { + const deadline = Date.now() + opts.timeoutMs; + const pollMs = opts.pollMs ?? 2000; + while (Date.now() < deadline) { + const status = await getJobStatus(clients, namespace, name); + if (status.phase === "Succeeded" || status.phase === "Failed") return status; + await sleep(pollMs); + } + throw new JobTimeoutError(namespace, name, opts.timeoutMs); +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** + * Job-backed conformance to SandboxOrchestrator. Plugin.ts imports THIS value + * (the swap point) — to use a different backend, swap this import for another + * module exposing a SandboxOrchestrator-shaped default export. + */ +export const jobOrchestrator: SandboxOrchestrator = { + claim: createJob, + getStatus: getJobStatus, + findPod: findPodForJob, + streamLogs: streamPodLogs, + release: deleteJob, + waitForCompletion: waitForJobCompletion, +}; diff --git a/packages/plugins/sandbox-providers/kubernetes/src/kube-client.ts b/packages/plugins/sandbox-providers/kubernetes/src/kube-client.ts new file mode 100644 index 00000000..d66a9fc4 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/kube-client.ts @@ -0,0 +1,44 @@ +import { + KubeConfig, + CoreV1Api, + BatchV1Api, + CustomObjectsApi, + NetworkingV1Api, + RbacAuthorizationV1Api, +} from "@kubernetes/client-node"; + +export interface CreateKubeConfigInput { + inCluster?: boolean; + kubeconfig?: string; +} + +export function createKubeConfig(input: CreateKubeConfigInput): KubeConfig { + const kc = new KubeConfig(); + if (input.inCluster) { + kc.loadFromCluster(); + return kc; + } + if (input.kubeconfig && input.kubeconfig.trim().length > 0) { + kc.loadFromString(input.kubeconfig); + return kc; + } + throw new Error("createKubeConfig requires either inCluster=true or a kubeconfig string"); +} + +export interface KubeClients { + core: CoreV1Api; + batch: BatchV1Api; + custom: CustomObjectsApi; + networking: NetworkingV1Api; + rbac: RbacAuthorizationV1Api; +} + +export function makeKubeClients(kc: KubeConfig): KubeClients { + return { + core: kc.makeApiClient(CoreV1Api), + batch: kc.makeApiClient(BatchV1Api), + custom: kc.makeApiClient(CustomObjectsApi), + networking: kc.makeApiClient(NetworkingV1Api), + rbac: kc.makeApiClient(RbacAuthorizationV1Api), + }; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/lease-lifecycle.ts b/packages/plugins/sandbox-providers/kubernetes/src/lease-lifecycle.ts new file mode 100644 index 00000000..87e18aeb --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/lease-lifecycle.ts @@ -0,0 +1,193 @@ +/** + * Resume + destroy lifecycle helpers for Kubernetes sandbox leases. + * + * Resume semantics: a lease is resumable only while its workload resource + * (Sandbox CR or Job) still exists and its pod is Running/Ready (or becomes + * Ready within a short bounded wait). Unlike Daytona — where a stopped sandbox + * can be started again by ID — Kubernetes pods are NOT restartable: once the + * pod backing a lease is gone or terminally failed, the lease can never be + * revived in place. That asymmetry is intentional; the plugin reports the + * lease as expired and the server falls back to a fresh acquireLease, which + * provisions a new pod. + * + * Destroy semantics: the forced cleanup path. Deletes every resource + * acquireLease created (Sandbox CR / Job, its pod, the per-run Secret), + * treating 404s as success so it is idempotent and safe to call against + * half-deleted leases. + */ + +import type { KubeClients } from "./kube-client.js"; +import { deleteJob, findPodForJob, getJobStatus } from "./job-orchestrator.js"; +import { + deleteSandboxCr, + findPodForSandbox, + waitForSandboxReady, +} from "./sandbox-cr-orchestrator.js"; + +/** True when a Kubernetes API error means "resource not found" (HTTP 404). */ +export function isKubeNotFoundError(err: unknown): boolean { + const code = (err as { code?: number; statusCode?: number }).code + ?? (err as { code?: number; statusCode?: number }).statusCode; + return code === 404; +} + +async function ignoreNotFound(promise: Promise): Promise { + try { + await promise; + } catch (err) { + if (!isKubeNotFoundError(err)) throw err; + } +} + +export type ResumeCheckResult = + | { resumable: true; podName: string | null; phase: "Pending" | "Running" } + | { resumable: false; reason: string }; + +export interface ResumeCheckInput { + namespace: string; + /** Workload resource name (Sandbox CR name or Job name) == providerLeaseId. */ + name: string; + backend: "sandbox-cr" | "job"; + /** Bounded wait for an existing Sandbox pod to report Ready. */ + readyTimeoutMs?: number; + pollMs?: number; +} + +/** + * Check whether the workload behind a lease is still alive and exec-able. + * Returns `resumable: false` (never throws "expected" states) when the + * resource is gone (404), terminally failed, terminating, or doesn't become + * Ready within the bounded wait — all of which mean the caller should fall + * back to a fresh acquireLease. + */ +export async function checkLeaseResumable( + clients: KubeClients, + input: ResumeCheckInput, +): Promise { + if (input.backend === "sandbox-cr") { + // Bounded wait for the Sandbox to report Ready. waitForSandboxReady fails + // fast on Failed/Terminating; a timeout means the pod never came up. None + // of those states are resumable — k8s pods cannot be restarted in place. + try { + await waitForSandboxReady(clients, input.namespace, input.name, { + timeoutMs: input.readyTimeoutMs ?? 30_000, + pollMs: input.pollMs ?? 1_000, + }); + } catch (err) { + if (isKubeNotFoundError(err)) { + return { resumable: false, reason: "Sandbox CR no longer exists" }; + } + return { + resumable: false, + reason: err instanceof Error ? err.message : String(err), + }; + } + + let podName: string | null; + try { + podName = await findPodForSandbox(clients, input.namespace, input.name); + } catch (err) { + // CR deleted between the readiness check and the pod lookup. + if (isKubeNotFoundError(err)) { + return { resumable: false, reason: "Sandbox CR no longer exists" }; + } + throw err; + } + if (!podName) { + return { + resumable: false, + reason: "Sandbox is Ready but no backing pod was found", + }; + } + + // Confirm the pod itself is Running and not being torn down — the CR + // status can lag pod deletion. + let pod: { metadata?: { deletionTimestamp?: unknown }; status?: { phase?: string } }; + try { + pod = await clients.core.readNamespacedPod({ + namespace: input.namespace, + name: podName, + }) as typeof pod; + } catch (err) { + if (isKubeNotFoundError(err)) { + return { resumable: false, reason: `Pod ${podName} no longer exists` }; + } + throw err; + } + const podPhase = pod.status?.phase; + const terminating = Boolean(pod.metadata?.deletionTimestamp); + if (podPhase !== "Running" || terminating) { + return { + resumable: false, + reason: `Pod ${podName} is ${terminating ? "terminating" : podPhase ?? "in an unknown phase"}`, + }; + } + return { resumable: true, podName, phase: "Running" }; + } + + // ── Job backend ─────────────────────────────────────────────────────────── + let status; + try { + status = await getJobStatus(clients, input.namespace, input.name); + } catch (err) { + if (isKubeNotFoundError(err)) { + return { resumable: false, reason: "Job no longer exists" }; + } + throw err; + } + if (status.phase === "Succeeded" || status.phase === "Failed") { + // Terminal Jobs cannot be re-run in place. + return { resumable: false, reason: `Job is ${status.phase}` }; + } + // Pending/Running Jobs are resumable: execute() waits for completion + // itself, so a not-yet-scheduled pod (podName null) is fine here. + const podName = await findPodForJob(clients, input.namespace, input.name); + return { + resumable: true, + podName, + phase: status.phase === "Running" ? "Running" : "Pending", + }; +} + +export interface DestroyLeaseInput { + namespace: string; + /** Workload resource name (Sandbox CR name or Job name) == providerLeaseId. */ + name: string; + backend: "sandbox-cr" | "job"; + podName: string | null; + secretName: string | null; +} + +/** + * Forcibly delete every resource acquireLease created for this lease. + * Workload first (its deletion cascades to the pod and, via ownerReferences, + * the per-run Secret in the normal case); then the pod and Secret explicitly + * so a wedged controller or broken ownerRef cannot strand them. Every delete + * treats 404 as success — destroy is idempotent. + */ +export async function destroyLeaseResources( + clients: KubeClients, + input: DestroyLeaseInput, +): Promise { + if (input.backend === "sandbox-cr") { + await ignoreNotFound(deleteSandboxCr(clients, input.namespace, input.name)); + } else { + await ignoreNotFound(deleteJob(clients, input.namespace, input.name)); + } + if (input.podName) { + await ignoreNotFound( + clients.core.deleteNamespacedPod({ + namespace: input.namespace, + name: input.podName, + }), + ); + } + if (input.secretName) { + await ignoreNotFound( + clients.core.deleteNamespacedSecret({ + namespace: input.namespace, + name: input.secretName, + }), + ); + } +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/manifest.ts b/packages/plugins/sandbox-providers/kubernetes/src/manifest.ts new file mode 100644 index 00000000..c10ab84f --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/manifest.ts @@ -0,0 +1,121 @@ +import type { PaperclipPluginManifestV1 } from "@paperclipai/plugin-sdk"; + +const PLUGIN_ID = "paperclip.kubernetes-sandbox-provider"; +const PLUGIN_VERSION = "0.1.0-alpha.1"; + +const manifest: PaperclipPluginManifestV1 = { + id: PLUGIN_ID, + apiVersion: 1, + version: PLUGIN_VERSION, + displayName: "Kubernetes Sandbox (alpha)", + description: + "Built on kubernetes-sigs/agent-sandbox (v1alpha1). ALPHA — expect breaking changes as the upstream CRD evolves. Falls back to stable batch/v1 Job mode for clusters without agent-sandbox installed. First-party Paperclip sandbox-provider plugin for Kubernetes.", + author: "Paperclip", + categories: ["automation"], + capabilities: ["environment.drivers.register"], + entrypoints: { + worker: "./dist/worker.js", + }, + environmentDrivers: [ + { + driverKey: "kubernetes", + kind: "sandbox_provider", + displayName: "Kubernetes", + description: + "Dispatches agent runs in per-tenant Kubernetes namespaces. Default backend (sandbox-cr, alpha) uses kubernetes-sigs/agent-sandbox for multi-command exec; fallback backend (job) uses stable batch/v1 Job for clusters without agent-sandbox installed.", + configSchema: { + type: "object", + properties: { + inCluster: { + type: "boolean", + description: + "When true, the plugin uses the in-pod ServiceAccount credentials. Requires paperclip-server to be running inside the target cluster.", + }, + kubeconfig: { + type: "string", + format: "secret-ref", + description: + "Inline kubeconfig YAML. Paste a kubeconfig or an existing Paperclip secret reference; pasted values are stored as company secrets.", + }, + namespacePrefix: { + type: "string", + description: "Prefix for the per-company tenant namespace (default: paperclip-).", + }, + companySlug: { + type: "string", + description: "Override the auto-derived company slug used in the tenant namespace name.", + }, + imageRegistry: { + type: "string", + description: "Override the default registry for agent runtime images (default: ghcr.io/paperclipai).", + }, + imageAllowList: { + type: "array", + items: { type: "string" }, + description: + "Glob patterns of allowed `target.imageOverride` values. Empty list = no override permitted.", + }, + imagePullSecrets: { + type: "array", + items: { type: "string" }, + description: "Names of pre-created Docker image pull secrets in the tenant namespace.", + }, + egressAllowFqdns: { + type: "array", + items: { type: "string" }, + description: + "Additional FQDNs to allow egress to from agent pods. Adapter-default FQDNs (e.g. api.anthropic.com) are added automatically.", + }, + egressAllowCidrs: { + type: "array", + items: { type: "string" }, + description: "Additional CIDRs to allow egress to from agent pods.", + }, + egressMode: { + type: "string", + enum: ["standard", "cilium"], + description: "Network policy mode. `cilium` enables FQDN-based egress filtering via CiliumNetworkPolicy.", + }, + runtimeClassName: { + type: "string", + description: + "Optional RuntimeClass for pod isolation (e.g. `kata-fc` for Firecracker-backed microVMs). Cluster must have the RuntimeClass installed.", + }, + serviceAccountAnnotations: { + type: "object", + additionalProperties: { type: "string" }, + description: + "Annotations applied to the per-tenant ServiceAccount (e.g. `eks.amazonaws.com/role-arn` for IRSA).", + }, + jobTtlSecondsAfterFinished: { + type: "integer", + minimum: 0, + description: "Seconds after a Job completes before it is garbage-collected (default: 900).", + }, + podActivityDeadlineSec: { + type: "integer", + minimum: 1, + description: "Hard ceiling on a single run's wall-clock time (default: 3600).", + }, + adapterType: { + type: "string", + description: + "The adapter type that Jobs in this environment will run (e.g. `claude_local`, `codex_local`). Defaults to `claude_local`. Each environment is bound to one adapter; create multiple environments for different adapters.", + }, + backend: { + type: "string", + enum: ["sandbox-cr", "job"], + description: + "sandbox-cr (default, alpha — requires kubernetes-sigs/agent-sandbox installed) | job (stable fallback — batch/v1 Job, one-shot entrypoint, no multi-command exec)", + }, + }, + anyOf: [ + { required: ["inCluster"] }, + { required: ["kubeconfig"] }, + ], + }, + }, + ], +}; + +export default manifest; diff --git a/packages/plugins/sandbox-providers/kubernetes/src/network-policy.ts b/packages/plugins/sandbox-providers/kubernetes/src/network-policy.ts new file mode 100644 index 00000000..4878a3b7 --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/network-policy.ts @@ -0,0 +1,132 @@ +export interface BuildNetworkPolicyInput { + namespace: string; + paperclipServerNamespace: string; + egressAllowCidrs: string[]; + /** + * Adapter-configured FQDNs (e.g. `api.anthropic.com`). Standard + * NetworkPolicy cannot express FQDNs natively — only Cilium can. + * When this list is non-empty AND no explicit `egressAllowCidrs` + * was provided, the standard NetworkPolicy falls back to "public + * IPv4 except RFC1918/link-local/loopback/multicast" so the + * configured FQDNs at least become reachable. This is broader + * than the operator probably wants — switch to `egressMode: + * "cilium"` for exact FQDN allow-listing in production. + */ + egressAllowFqdns?: string[]; +} + +/** + * IPv4 ranges to carve out of `0.0.0.0/0` when we apply the + * public-internet fallback. Keeps agent pods from reaching cluster + * internals (RFC1918), node link-local + AWS IMDS (169.254.0.0/16), + * cluster loopback (127.0.0.0/8), CGNAT (100.64.0.0/10), this-network + * (0.0.0.0/8), and multicast (224.0.0.0/4). + */ +const PRIVATE_AND_LINK_LOCAL_EXCEPT_CIDRS = [ + "0.0.0.0/8", + "10.0.0.0/8", + "100.64.0.0/10", + "127.0.0.0/8", + "169.254.0.0/16", + "172.16.0.0/12", + "192.168.0.0/16", + "224.0.0.0/4", +]; + +// Design note: the deny-all baseline blocks all ingress to agent pods. +// Paperclip-server does NOT push to agent pods — the agent shim makes +// outbound calls to paperclip-server via the egress allow-list (port 3100). +// This pull/callback model means no ingress rule is needed. If a future +// feature requires server→agent push (e.g. forced shutdown, live exec), +// add a targeted ingress rule here scoped to the paperclip-server pod +// selector. +export function buildNetworkPolicyManifests(input: BuildNetworkPolicyInput): Record[] { + const denyAll = { + apiVersion: "networking.k8s.io/v1", + kind: "NetworkPolicy", + metadata: { + name: "paperclip-deny-all", + namespace: input.namespace, + labels: { "paperclip.io/managed-by": "paperclip-k8s-plugin" }, + }, + spec: { + podSelector: {}, + policyTypes: ["Ingress", "Egress"], + }, + }; + + const egressAllow: Record = { + apiVersion: "networking.k8s.io/v1", + kind: "NetworkPolicy", + metadata: { + name: "paperclip-egress-allow", + namespace: input.namespace, + labels: { "paperclip.io/managed-by": "paperclip-k8s-plugin" }, + }, + spec: { + podSelector: { matchLabels: { "paperclip.io/role": "agent" } }, + policyTypes: ["Egress"], + egress: [ + { + to: [ + { + namespaceSelector: { matchLabels: { "kubernetes.io/metadata.name": "kube-system" } }, + podSelector: { matchLabels: { "k8s-app": "kube-dns" } }, + }, + ], + ports: [ + { protocol: "UDP", port: 53 }, + { protocol: "TCP", port: 53 }, + ], + }, + { + to: [ + { + namespaceSelector: { matchLabels: { "kubernetes.io/metadata.name": input.paperclipServerNamespace } }, + podSelector: { matchLabels: { app: "paperclip-server" } }, + }, + ], + ports: [{ protocol: "TCP", port: 3100 }], + }, + // NOTE: operator-supplied CIDRs are intentionally NOT port-scoped — + // operators may need them for non-HTTP services (e.g. private VCS + // mirrors, S3 endpoints, internal artifact registries). Operators + // should keep CIDRs as specific as possible. For HTTP/HTTPS-only + // LLM endpoints, the public-IPv4 fallback below is port-scoped + // (TCP 80/443). + ...input.egressAllowCidrs.map((cidr) => ({ + to: [{ ipBlock: { cidr } }], + })), + // Standard-NetworkPolicy fallback for FQDN-based egress. If the + // adapter requires FQDNs (e.g. api.anthropic.com) and the + // operator didn't supply explicit CIDRs, allow public IPv4 with + // private/link-local/loopback/multicast carved out. This makes + // the default `egressMode: "standard"` config functional out of + // the box for cloud LLM APIs without inadvertently exposing + // cluster internals or link-local metadata endpoints. Operators + // who want exact FQDN enforcement should use `egressMode: "cilium"`. + ...(input.egressAllowCidrs.length === 0 && + (input.egressAllowFqdns?.length ?? 0) > 0 + ? [ + { + to: [ + { + ipBlock: { + cidr: "0.0.0.0/0", + except: PRIVATE_AND_LINK_LOCAL_EXCEPT_CIDRS, + }, + }, + ], + ports: [ + { protocol: "TCP", port: 443 }, + { protocol: "TCP", port: 80 }, + ], + }, + ] + : []), + ], + }, + }; + + return [denyAll, egressAllow]; +} diff --git a/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts new file mode 100644 index 00000000..8b89501a --- /dev/null +++ b/packages/plugins/sandbox-providers/kubernetes/src/plugin.ts @@ -0,0 +1,864 @@ +import { randomBytes } from "node:crypto"; +import { definePlugin } from "@paperclipai/plugin-sdk"; +import type { + PluginEnvironmentAcquireLeaseParams, + PluginEnvironmentDestroyLeaseParams, + PluginEnvironmentExecuteParams, + PluginEnvironmentExecuteResult, + PluginEnvironmentLease, + PluginEnvironmentProbeParams, + PluginEnvironmentProbeResult, + PluginEnvironmentRealizeWorkspaceParams, + PluginEnvironmentRealizeWorkspaceResult, + PluginEnvironmentReleaseLeaseParams, + PluginEnvironmentResumeLeaseParams, + PluginEnvironmentValidateConfigParams, + PluginEnvironmentValidationResult, +} from "@paperclipai/plugin-sdk"; +import { + kubernetesProviderConfigSchema, + type KubernetesProviderConfig, + type KubernetesLeaseMetadata, +} from "./types.js"; +import { createKubeConfig, makeKubeClients } from "./kube-client.js"; +import { getAdapterDefaults, buildAdapterEnv, resolveRunAdapterType } from "./adapter-defaults.js"; +import { resolveImage } from "./image-allowlist.js"; +import { buildJobManifest } from "./pod-spec-builder.js"; +import { buildSandboxCrManifest } from "./sandbox-cr-builder.js"; +import { ensureTenant } from "./tenant-orchestrator.js"; +import { createPerRunSecret } from "./secret-manager.js"; +import { FastUploadInterceptor } from "./upload-interceptor.js"; +import { jobOrchestrator, JobTimeoutError } from "./job-orchestrator.js"; +import { + sandboxCrOrchestrator, + SandboxCrTimeoutError, +} from "./sandbox-cr-orchestrator.js"; +import { execInPod, wrapCommandWithEnv } from "./pod-exec.js"; +import { checkLeaseResumable, destroyLeaseResources } from "./lease-lifecycle.js"; +import { + deriveCompanySlug, + deriveNamespaceName, + newRunUlidDns, + paperclipLabels, +} from "./utils.js"; + +// The namespace paperclip-server itself runs in. Used when building +// NetworkPolicy manifests so the tenant namespace allows inbound traffic +// from the server pod. +const PAPERCLIP_SERVER_NAMESPACE = "paperclip"; + +// Name of the ServiceAccount created inside each tenant namespace by ensureTenant. +const TENANT_SERVICE_ACCOUNT = "paperclip-tenant-sa"; + +// Resource quota defaults applied to every tenant namespace (tunable via +// config in a future iteration). +const DEFAULT_RESOURCE_QUOTA = { + pods: "20", + requestsCpu: "10", + requestsMemory: "20Gi", + limitsCpu: "20", + limitsMemory: "40Gi", +}; + +function deriveTenantNamespace(config: KubernetesProviderConfig, companyId: string): string { + // TODO: future versions could thread companyName through AcquireLeaseParams + // to get a friendlier slug (e.g. "acme-corp") instead of the UUID-derived one. + const slug = config.companySlug ?? deriveCompanySlug(companyId); + return deriveNamespaceName(config.namespacePrefix, slug); +} + +function generateBootstrapToken(): string { + // TODO: tighten once the agent runtime shim (companion images PR) lands its + // callback auth scheme; paperclip-server's callback auth is out of scope for + // this plugin. For now this per-run random token is stored in the per-run + // Secret and read by the runtime image entrypoint for initial registration. + return randomBytes(32).toString("hex"); +} + +// One FastUploadInterceptor instance per active lease. Scoping per lease +// prevents `releaseLease` from wiping in-flight upload buffers belonging to +// other concurrent leases — a single shared singleton would do exactly that +// on `reset()`. The Map is keyed by `providerLeaseId`; entries are lazily +// created in `onEnvironmentExecute` and removed in `onEnvironmentReleaseLease`. +const uploadInterceptorsByLease = new Map(); + +function getOrCreateUploadInterceptor(leaseId: string): FastUploadInterceptor { + let interceptor = uploadInterceptorsByLease.get(leaseId); + if (!interceptor) { + interceptor = new FastUploadInterceptor(); + uploadInterceptorsByLease.set(leaseId, interceptor); + } + return interceptor; +} + +// In-memory cache of sandbox CR names we've already observed reaching the +// Ready condition during the current plugin-worker lifetime. The k8s +// sandbox-cr lifecycle means once a Sandbox pod is Running, subsequent +// execs into it don't need another readiness poll — saves one +// `getNamespacedCustomObject` round-trip per exec, which adds up across +// dozens of sequential exec calls in a typical adapter workflow. +// On worker restart this resets, which is fine: the first exec on each +// lease then re-confirms readiness from scratch. +const readySandboxesByLease = new Set(); + +// How long onEnvironmentResumeLease waits for an existing Sandbox pod to +// report Ready before declaring the lease non-resumable. Deliberately short: +// this is a liveness check on an already-provisioned pod, not a fresh +// provision — if the pod isn't (almost) up, falling back to acquireLease is +// faster and more reliable than waiting. +const RESUME_READY_TIMEOUT_MS = 30_000; +const RESUME_READY_POLL_MS = 1_000; + +const plugin = definePlugin({ + async setup(ctx) { + ctx.logger.info("Kubernetes sandbox provider plugin ready"); + }, + + async onHealth() { + return { status: "ok", message: "Kubernetes sandbox provider plugin healthy" }; + }, + + async onEnvironmentValidateConfig( + params: PluginEnvironmentValidateConfigParams, + ): Promise { + const parsed = kubernetesProviderConfigSchema.safeParse(params.config); + if (!parsed.success) { + return { + ok: false, + errors: parsed.error.issues.map((i) => i.message), + }; + } + const warnings: string[] = []; + const cfg = parsed.data; + const adapterDefaults = getAdapterDefaults(cfg.adapterType, cfg.adapters); + const totalFqdns = [...adapterDefaults.allowFqdns, ...cfg.egressAllowFqdns]; + if (cfg.egressMode === "standard" && totalFqdns.length > 0) { + if (cfg.egressAllowCidrs.length === 0) { + warnings.push( + `egressMode=standard cannot enforce FQDN-based egress rules (Kubernetes NetworkPolicy is CIDR-only). To keep the configured FQDNs reachable (${totalFqdns.join(", ")}) without operator intervention, the plugin will allow public IPv4 egress on TCP 80/443 with private/link-local/loopback/multicast ranges excluded. This is broader than exact FQDN allow-listing — switch egressMode to "cilium" (requires Cilium CNI) for precise enforcement, or set egressAllowCidrs explicitly to override the fallback.`, + ); + } else { + warnings.push( + `egressMode=standard cannot enforce FQDN-based egress rules. The following FQDNs are reachable only via the operator-supplied egressAllowCidrs: ${totalFqdns.join(", ")}. Switch egressMode to "cilium" (requires Cilium CNI) for exact FQDN allow-listing.`, + ); + } + } + return { ok: true, normalizedConfig: cfg as Record, warnings: warnings.length > 0 ? warnings : undefined }; + }, + + async onEnvironmentProbe( + params: PluginEnvironmentProbeParams, + ): Promise { + const parsed = kubernetesProviderConfigSchema.safeParse(params.config); + if (!parsed.success) { + return { + ok: false, + summary: "Invalid Kubernetes provider configuration.", + metadata: { + errors: parsed.error.issues.map((i) => i.message), + }, + }; + } + const config = parsed.data; + const namespace = deriveTenantNamespace(config, params.companyId); + + try { + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + // Reachability check: list pods in the tenant namespace. If the namespace + // doesn't exist yet this will throw a 404 which we treat as "reachable + // but namespace not provisioned" — still a successful probe. + try { + await clients.core.listNamespacedPod({ namespace }); + } catch (err) { + const code = (err as { code?: number; statusCode?: number }).code + ?? (err as { code?: number; statusCode?: number }).statusCode; + if (code !== 404) throw err; + // 404 means namespace doesn't exist yet — cluster is reachable. + } + return { + ok: true, + summary: `Kubernetes cluster reachable. Tenant namespace: ${namespace}.`, + metadata: { namespace, provider: "kubernetes" }, + }; + } catch (err) { + return { + ok: false, + summary: "Kubernetes cluster probe failed.", + metadata: { + namespace, + provider: "kubernetes", + error: err instanceof Error ? err.message : String(err), + }, + }; + } + }, + + async onEnvironmentAcquireLease( + // `adapterType` is an optional per-run hint the server may pass once the + // SDK lease params grow that field (companion server-integration PR). The + // plugin works without it: absent means "use the environment's configured + // default adapter", so it stays compatible with the current SDK. + params: PluginEnvironmentAcquireLeaseParams & { adapterType?: string }, + ): Promise { + const config = kubernetesProviderConfigSchema.parse(params.config); + const namespace = deriveTenantNamespace(config, params.companyId); + + // The adapter for THIS run is the agent's adapter (params.adapterType) when + // supplied, so one environment can serve mixed harnesses; otherwise fall back + // to the environment's configured default adapter. getAdapterDefaults validates + // it is a registered adapter (throws otherwise), so a curated-out adapter fails + // the lease as before. + const effectiveAdapterType = resolveRunAdapterType(params.adapterType, config.adapterType); + + // Emit a runtime warning if FQDNs are configured but egressMode=standard + // cannot enforce them. Mirrors the validateConfig warning so operators see + // it in paperclip-server logs even if they missed the validation step. + const adapterDefaultsForWarn = getAdapterDefaults(effectiveAdapterType, config.adapters); + const totalFqdnsForWarn = [...adapterDefaultsForWarn.allowFqdns, ...config.egressAllowFqdns]; + if (config.egressMode === "standard" && totalFqdnsForWarn.length > 0) { + if (config.egressAllowCidrs.length === 0) { + console.warn( + `[plugin-kubernetes] egressMode=standard cannot enforce FQDN-based egress rules; falling back to public-IPv4 (TCP 80/443) with private/link-local ranges excluded so the configured FQDNs (${totalFqdnsForWarn.join(", ")}) remain reachable. Switch egressMode to "cilium" for exact FQDN allow-listing.`, + ); + } else { + console.warn( + `[plugin-kubernetes] egressMode=standard cannot enforce FQDN-based egress rules. The following FQDNs are reachable only via operator-supplied egressAllowCidrs: ${totalFqdnsForWarn.join(", ")}. Switch egressMode to "cilium" for exact FQDN allow-listing.`, + ); + } + } + + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + + // Ensure the tenant namespace and all its RBAC / network policy resources + // exist before we try to create the Job. + const adapterDefaults = getAdapterDefaults(effectiveAdapterType, config.adapters); + + await ensureTenant(clients, { + namespace, + companyId: params.companyId, + paperclipServerNamespace: PAPERCLIP_SERVER_NAMESPACE, + serviceAccountAnnotations: config.serviceAccountAnnotations, + egressMode: config.egressMode, + egressAllowFqdns: [...adapterDefaults.allowFqdns, ...config.egressAllowFqdns], + egressAllowCidrs: config.egressAllowCidrs, + resourceQuota: DEFAULT_RESOURCE_QUOTA, + }); + + const jobName = `pc-${newRunUlidDns()}`; + const secretName = `${jobName}-env`; + + // TODO: use params.runId as stand-in for agentId in labels; future + // versions will have a dedicated agentId on AcquireLeaseParams. + const labels = paperclipLabels({ + runId: params.runId, + agentId: params.runId, + companyId: params.companyId, + adapterType: effectiveAdapterType, + }); + + const image = resolveImage( + { imageOverride: null }, + adapterDefaults, + { imageAllowList: config.imageAllowList, imageRegistry: config.imageRegistry }, + ); + + // Pick the orchestrator and build the appropriate manifest based on backend. + const isSandboxCrBackend = config.backend === "sandbox-cr"; + const orchestrator = isSandboxCrBackend ? sandboxCrOrchestrator : jobOrchestrator; + + const manifest = isSandboxCrBackend + ? buildSandboxCrManifest({ + namespace, + sandboxName: jobName, + adapterType: effectiveAdapterType, + image, + envSecretName: secretName, + serviceAccountName: TENANT_SERVICE_ACCOUNT, + labels, + resources: config.defaultResources ?? {}, + runtimeClassName: config.runtimeClassName, + imagePullSecrets: config.imagePullSecrets, + }) + : buildJobManifest({ + namespace, + jobName, + adapterType: effectiveAdapterType, + image, + envSecretName: secretName, + serviceAccountName: TENANT_SERVICE_ACCOUNT, + labels, + resources: config.defaultResources ?? {}, + runtimeClassName: config.runtimeClassName, + activeDeadlineSec: config.podActivityDeadlineSec, + ttlSecondsAfterFinished: config.jobTtlSecondsAfterFinished, + imagePullSecrets: config.imagePullSecrets, + }); + + const { uid: ownerUid } = await orchestrator.claim(clients, namespace, manifest); + + // defaultEnv (non-secret base, e.g. the inference base URL) is layered first; + // the process-env secrets named by envKeys override it. + const adapterEnv = buildAdapterEnv(adapterDefaults); + const bootstrapToken = generateBootstrapToken(); + + // Secret ownerRef: for job backend, the Job owns the Secret (cascade delete). + // For sandbox-cr backend, the Sandbox CR owns the Secret. + // NOTE: For sandbox-cr, if the Secret outlives the Sandbox due to a cluster + // quirk, the release() call will still clean it up via namespace GC or + // explicit delete in a future iteration. + await createPerRunSecret(clients, { + namespace, + secretName, + runId: params.runId, + ownerKind: isSandboxCrBackend ? "Sandbox" : "Job", + ownerApiVersion: isSandboxCrBackend ? "agents.x-k8s.io/v1alpha1" : "batch/v1", + ownerName: jobName, + ownerUid, + bootstrapToken, + adapterEnv, + }); + + const podName = await orchestrator.findPod(clients, namespace, jobName); + + const leaseMetadata: KubernetesLeaseMetadata = { + namespace, + jobName, + podName, + secretName, + phase: "Pending", + backend: config.backend, + }; + + return { + providerLeaseId: jobName, + metadata: leaseMetadata as unknown as Record, + }; + }, + + async onEnvironmentResumeLease( + params: PluginEnvironmentResumeLeaseParams, + ): Promise { + const config = kubernetesProviderConfigSchema.parse(params.config); + const namespace = + typeof params.leaseMetadata?.namespace === "string" + ? params.leaseMetadata.namespace + : deriveTenantNamespace(config, params.companyId); + const leaseBackend = + typeof params.leaseMetadata?.backend === "string" + ? (params.leaseMetadata.backend as "sandbox-cr" | "job") + : config.backend; + // acquireLease names the per-run Secret `${jobName}-env` and uses jobName + // as the providerLeaseId, so the suffix fallback reconstructs it exactly. + const secretName = + typeof params.leaseMetadata?.secretName === "string" + ? params.leaseMetadata.secretName + : `${params.providerLeaseId}-env`; + + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + + const check = await checkLeaseResumable(clients, { + namespace, + name: params.providerLeaseId, + backend: leaseBackend, + readyTimeoutMs: RESUME_READY_TIMEOUT_MS, + pollMs: RESUME_READY_POLL_MS, + }); + + if (!check.resumable) { + // Kubernetes pods are NOT restartable the way Daytona sandboxes are: a + // stopped Daytona sandbox can be started again by ID, but a k8s pod that + // is gone or terminally failed can never be revived in place. Gone = not + // resumable, by design. Returning providerLeaseId: null tells the server + // the lease expired so it falls back to a fresh acquireLease. + return { + providerLeaseId: null, + metadata: { expired: true, reason: check.reason }, + }; + } + + // A resumed lease starts with clean per-lease state: drop any stale upload + // interceptor buffers a previous run on this lease may have left behind. + uploadInterceptorsByLease.delete(params.providerLeaseId); + if (leaseBackend === "sandbox-cr") { + // We just observed the Sandbox pod Ready, so the first exec on the + // resumed lease can skip its readiness poll. + readySandboxesByLease.add(params.providerLeaseId); + } + + const leaseMetadata: KubernetesLeaseMetadata = { + namespace, + jobName: params.providerLeaseId, + podName: check.podName, + secretName, + phase: check.phase, + backend: leaseBackend, + }; + + return { + providerLeaseId: params.providerLeaseId, + metadata: { + ...leaseMetadata, + resumedLease: true, + } as unknown as Record, + }; + }, + + async onEnvironmentRealizeWorkspace( + params: PluginEnvironmentRealizeWorkspaceParams, + ): Promise { + // The agent pod already has /workspace mounted as an emptyDir at pod + // scheduling time (see pod-spec-builder). Nothing to provision here — + // we just hand back the cwd. Honor a caller-supplied remotePath if set. + const cwd = + params.workspace.remotePath && params.workspace.remotePath.trim().length > 0 + ? params.workspace.remotePath.trim() + : "/workspace"; + return { + cwd, + metadata: { + provider: "kubernetes", + remoteCwd: cwd, + }, + }; + }, + + async onEnvironmentReleaseLease( + params: PluginEnvironmentReleaseLeaseParams, + ): Promise { + if (!params.providerLeaseId) return; + const config = kubernetesProviderConfigSchema.parse(params.config); + const namespace = + typeof params.leaseMetadata?.namespace === "string" + ? params.leaseMetadata.namespace + : deriveTenantNamespace(config, params.companyId); + + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + + const leaseBackend = + typeof params.leaseMetadata?.backend === "string" + ? (params.leaseMetadata.backend as "sandbox-cr" | "job") + : config.backend; + const releaseOrchestrator = + leaseBackend === "sandbox-cr" ? sandboxCrOrchestrator : jobOrchestrator; + + // Drop the FastUploadInterceptor associated with THIS lease (only). + // Each lease has its own interceptor instance via uploadInterceptorsByLease, + // so unrelated concurrent leases keep their in-flight buffers intact. + uploadInterceptorsByLease.delete(params.providerLeaseId); + readySandboxesByLease.delete(params.providerLeaseId); + + try { + await releaseOrchestrator.release(clients, namespace, params.providerLeaseId); + } catch (err) { + // If the resource is already gone (404), that's fine. + const code = (err as { code?: number; statusCode?: number }).code + ?? (err as { code?: number; statusCode?: number }).statusCode; + if (code !== 404) throw err; + } + }, + + async onEnvironmentDestroyLease( + params: PluginEnvironmentDestroyLeaseParams, + ): Promise { + if (!params.providerLeaseId) return; + const config = kubernetesProviderConfigSchema.parse(params.config); + const namespace = + typeof params.leaseMetadata?.namespace === "string" + ? params.leaseMetadata.namespace + : deriveTenantNamespace(config, params.companyId); + const leaseBackend = + typeof params.leaseMetadata?.backend === "string" + ? (params.leaseMetadata.backend as "sandbox-cr" | "job") + : config.backend; + const secretName = + typeof params.leaseMetadata?.secretName === "string" + ? params.leaseMetadata.secretName + : `${params.providerLeaseId}-env`; + const podName = + typeof params.leaseMetadata?.podName === "string" && + params.leaseMetadata.podName.length > 0 + ? params.leaseMetadata.podName + : null; + + // Clear per-lease in-memory state up front, regardless of what the + // cluster says — the lease is dead either way. + uploadInterceptorsByLease.delete(params.providerLeaseId); + readySandboxesByLease.delete(params.providerLeaseId); + + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + + // Forcibly delete everything acquireLease created (Sandbox CR / Job, pod, + // per-run Secret). 404s are success — destroy must be idempotent. + await destroyLeaseResources(clients, { + namespace, + name: params.providerLeaseId, + backend: leaseBackend, + podName, + secretName, + }); + }, + + async onEnvironmentExecute( + params: PluginEnvironmentExecuteParams, + ): Promise { + const { lease, timeoutMs } = params; + + if (!lease.providerLeaseId) { + return { + exitCode: 1, + timedOut: false, + stdout: "", + stderr: "No provider lease ID available for execution.", + }; + } + + const config = kubernetesProviderConfigSchema.parse(params.config); + const namespace = + typeof lease.metadata?.namespace === "string" + ? lease.metadata.namespace + : deriveTenantNamespace(config, params.companyId); + + // Determine which backend this lease was created with. + const leaseBackend = + typeof lease.metadata?.backend === "string" + ? (lease.metadata.backend as "sandbox-cr" | "job") + : config.backend; + + const kc = createKubeConfig({ + inCluster: config.inCluster, + kubeconfig: config.kubeconfig, + }); + const clients = makeKubeClients(kc); + + const effectiveTimeoutMs = + typeof timeoutMs === "number" && timeoutMs > 0 + ? timeoutMs + : config.podActivityDeadlineSec * 1000; + + if (leaseBackend === "sandbox-cr") { + // ── Sandbox-CR backend ────────────────────────────────────────────────── + // 1. Ensure the Sandbox pod is Ready (wait only on first exec for this lease). + // 2. Exec the command into the running pod. + // 3. Return exec result directly (no log scraping needed). + + let podName = + typeof lease.metadata?.podName === "string" && lease.metadata.podName + ? lease.metadata.podName + : null; + + // Skip the readiness poll if we've already observed this Sandbox CR + // reaching Ready during this worker's lifetime. See readySandboxesByLease + // declaration for rationale. + const podAlreadyKnownReady = readySandboxesByLease.has(lease.providerLeaseId); + + // The caller's timeout is a budget for the WHOLE execute call: readiness + // wait + exec must share it, or the first exec on a fresh lease could + // block for up to twice the requested timeout. + const executeStartedAt = Date.now(); + + if (!podAlreadyKnownReady) { + try { + await sandboxCrOrchestrator.waitForCompletion( + clients, + namespace, + lease.providerLeaseId, + { timeoutMs: effectiveTimeoutMs, pollMs: 2000 }, + ); + readySandboxesByLease.add(lease.providerLeaseId); + } catch (err) { + if (err instanceof SandboxCrTimeoutError) { + return { + exitCode: null, + timedOut: true, + stdout: "", + stderr: `Sandbox pod did not become Ready within ${effectiveTimeoutMs}ms`, + metadata: { + provider: "kubernetes", + backend: "sandbox-cr", + namespace, + sandboxName: lease.providerLeaseId, + }, + }; + } + throw err; + } + } + + // Resolve pod name (may now be populated in Sandbox status). + if (!podName) { + podName = await sandboxCrOrchestrator.findPod( + clients, + namespace, + lease.providerLeaseId, + ); + } + + if (!podName) { + return { + exitCode: 1, + timedOut: false, + stdout: "", + stderr: "Sandbox pod is Ready but podName could not be resolved.", + metadata: { + provider: "kubernetes", + backend: "sandbox-cr", + namespace, + sandboxName: lease.providerLeaseId, + }, + }; + } + + // Build the command to exec. The adapter passes shell invocations as + // `command: "sh", args: ["-c", "