feat(server): kubernetes execution integration for sandbox-provider plugins (stage 2/3) (#7938)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The execution subsystem runs those agents in environments (local, ssh, sandbox), and sandbox-provider plugins let an environment materialize per-run sandboxes > - Stage 1 (#5790) contributed a first-party Kubernetes sandbox-provider plugin, but the server core has no way to adopt it operationally: no per-run adapter selection, no way to force an instance onto sandboxed execution, no declarative adapter/model configuration, and the plugin must be installed by hand > - Without this, a multi-tenant or security-conscious deployment cannot guarantee that agent runs never execute on the host, and a single environment cannot serve agents with different harnesses > - This pull request adds the server + SDK integration: per-run adapterType on the lease protocol, an env-gated forced-Kubernetes execution policy with provisioning and a per-run allowlist guard, a declarative adapter registry and model list, in-cluster env passthrough for sandbox plugin workers, fail-safe auto-install of the bundled plugin, and the matching UI affordance > - The benefit is that sandbox-provider plugins become fully usable for Kubernetes execution: operators configure everything via environment variables and GitOps, while self-hosters who set none of the variables see exactly the behavior they have today ## Linked Issues or Issue Description Refs #5790 (stage 1 of 3: the Kubernetes sandbox-provider plugin package). No existing issue. Feature description: the server core lacks the integration seams to operate a sandbox-provider plugin as the mandatory execution path of an instance. This PR is stage 2 of 3 of the staged Kubernetes contribution; stage 3 will contribute the agent runtime images and their build pipeline. ## What Changed One line per piece: - `packages/plugins/sdk/protocol.ts`: optional `adapterType` on `PluginEnvironmentAcquireLeaseParams` so a provider can select the runtime image per run; existing providers simply ignore it - `server/services/environment-runtime.ts` + `environment-run-orchestrator.ts`: thread the agent's adapter type into both lease-acquiring drivers, including the heartbeat path (the two call sites have historically drifted, hence the pinned test) - `server/services/environments.ts`: `ensureKubernetesEnvironment` / `findKubernetesEnvironment`, an idempotent managed Kubernetes environment per company, identified by a metadata marker and refreshed (not recreated) on config change; `timeoutMs` rides on the config for slow cold-start leases - `server/services/execution-allowlist.ts`: pure (driver, provider, policy) -> allow/deny guard; `executionMode=kubernetes` only allows the kubernetes sandbox provider - `server/services/execution-policy-bootstrap.ts` + startup hook in `server/index.ts`: parse `PAPERCLIP_EXECUTION_MODE` / `PAPERCLIP_K8S_*`, persist `executionMode` into instance general settings, and provision the managed environment for every company; fails loud on misconfiguration - `server/services/heartbeat.ts`: when the policy forces Kubernetes, pin run selection to the managed environment (also overriding any persisted workspace environment id), refuse to fall back to local, and re-check the actually acquired environment against the allowlist as defense in depth - `server/services/adapter-registry-bootstrap.ts` + shared `AdapterRegistryEntry` type/validator: declarative `PAPERCLIP_ADAPTERS` registry (inline JSON or file) that reconciles adapter availability at startup and rides on the Kubernetes environment config - `server/services/adapter-models-env.ts` + `adapters/registry.ts`: `PAPERCLIP_ADAPTER_MODELS` lets an operator declare picker model lists the server cannot CLI-discover - `server/services/plugin-loader.ts`: pass `KUBERNETES_SERVICE_HOST/PORT(_HTTPS)` through to plugin workers that register environment drivers, so in-cluster API clients can be constructed; all other host env stays stripped - `server/app.ts`: fail-safe auto-install of the bundled kubernetes plugin at boot; no-ops when the bundle is absent and never blocks startup on error - `packages/shared` types/validators: `InstanceExecutionMode` on general settings (optional, strict schema) - `ui/lib/forced-kubernetes-environment.ts` + `AgentConfigForm`: when the policy is active, show a read-only Kubernetes environment instead of the environment picker and default new agents onto the managed environment - Tests for every new module plus the adapterType pin in `heartbeat-plugin-environment` and the managed-environment lifecycle in `environment-service` Everything is gated: with `PAPERCLIP_EXECUTION_MODE`, `PAPERCLIP_ADAPTERS`, and `PAPERCLIP_ADAPTER_MODELS` unset (and no bundled plugin present), every code path reduces to current behavior. The per-run `adapterType` is an optional SDK parameter that existing providers ignore. ## Verification - `cd server && npx tsc --noEmit`: clean (0 errors); `ui` typecheck also clean - Targeted suites all green (11 files, 90 tests): `npx vitest run server/src/__tests__/heartbeat-plugin-environment.test.ts server/src/__tests__/environment-service.test.ts server/src/__tests__/environment-runtime.test.ts server/src/__tests__/environment-run-orchestrator.test.ts server/src/__tests__/plugin-database.test.ts server/src/services/execution-policy-bootstrap.test.ts server/src/services/execution-allowlist.test.ts server/src/services/adapter-registry-bootstrap.test.ts server/src/services/adapter-registry-bootstrap.reconcile.test.ts server/src/services/adapter-models-env.test.ts packages/shared/src/validators/adapter-registry.test.ts` - `npx vitest run ui/src/components/AgentConfigForm.test.ts`: green (6 tests) - Full `npx vitest run server/src/__tests__`: 2323 passed, 1 skipped; the only failures (heartbeat-process-recovery pid-retry, workspace-runtime symbolic-ref/git tests) reproduce identically on pristine `master` in the same environment, so they are machine-environment issues unrelated to this change; `server-startup-feedback-export` needed its `services/index.js` mock extended with the new export and is green - This integration has been running in production on a hosted multi-tenant deployment, where it executes agent runs across five different harnesses through the stage 1 plugin ## Risks - Low for existing deployments: every behavior is env-gated and the defaults preserve current semantics; the auto-install block is wrapped fail-safe and skips silently when the plugin bundle is absent - `executionMode` is a new optional field on a strict zod schema; absent input normalizes exactly as before - The forced policy intentionally fails runs loudly (rather than falling back to local) when no managed Kubernetes environment exists; this only affects instances that explicitly set `PAPERCLIP_EXECUTION_MODE=kubernetes` ## Model Used Claude Opus 4.8 (claude-opus-4-8, 1M context), extended thinking, agentic tool use via Claude Code. ## UI screenshots The UI change is a new read-only "Execution" section in `AgentConfigForm`, shown only when the instance execution policy forces Kubernetes (`executionMode=kubernetes`); there is no "before" state for it (the section did not exist, and instances without the forced policy render the existing picker unchanged). Captured from the new Storybook stories added in this PR (`Product/Agent Management`): Managed Kubernetes environment present (read-only display, no local/SSH picker):  No managed environment available yet (warning notice, no silent local fallback):  ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jannes Stubbemann
committed by
GitHub
GitHub
parent
05ab45225a
commit
4ad94d0bde
@@ -1,4 +1,4 @@
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { models as claudeFallbackModels } from "@paperclipai/adapter-claude-local";
|
||||
import { resetClaudeModelsCacheForTests } from "@paperclipai/adapter-claude-local/server";
|
||||
import { models as codexFallbackModels } from "@paperclipai/adapter-codex-local";
|
||||
@@ -220,4 +220,64 @@ describe("adapter model listing", () => {
|
||||
expect(first.some((model) => model.id === "composer-1")).toBe(true);
|
||||
});
|
||||
|
||||
describe("PAPERCLIP_ADAPTER_MODELS declared models", () => {
|
||||
afterEach(() => {
|
||||
delete process.env.PAPERCLIP_ADAPTER_MODELS;
|
||||
});
|
||||
|
||||
it("prefers declared env models over adapter discovery", async () => {
|
||||
process.env.PAPERCLIP_ADAPTER_MODELS = JSON.stringify({
|
||||
opencode_local: [
|
||||
{ id: "tensorix/deepseek/deepseek-chat-v3.1", label: "DeepSeek v3.1" },
|
||||
{ id: "tensorix/z-ai/glm-4.7" },
|
||||
],
|
||||
});
|
||||
|
||||
const models = await listAdapterModels("opencode_local");
|
||||
|
||||
expect(models).toEqual([
|
||||
{ id: "tensorix/deepseek/deepseek-chat-v3.1", label: "DeepSeek v3.1" },
|
||||
{ id: "tensorix/z-ai/glm-4.7", label: "tensorix/z-ai/glm-4.7" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("observes env changes between calls (memo keyed by raw env value)", async () => {
|
||||
process.env.PAPERCLIP_ADAPTER_MODELS = JSON.stringify({
|
||||
opencode_local: [{ id: "model-a" }],
|
||||
});
|
||||
expect(await listAdapterModels("opencode_local")).toEqual([
|
||||
{ id: "model-a", label: "model-a" },
|
||||
]);
|
||||
|
||||
process.env.PAPERCLIP_ADAPTER_MODELS = JSON.stringify({
|
||||
opencode_local: [{ id: "model-b" }],
|
||||
});
|
||||
expect(await listAdapterModels("opencode_local")).toEqual([
|
||||
{ id: "model-b", label: "model-b" },
|
||||
]);
|
||||
});
|
||||
|
||||
it("fails soft on malformed values: falls back to adapter models instead of throwing", async () => {
|
||||
process.env.PAPERCLIP_ADAPTER_MODELS = "{not json";
|
||||
process.env.PAPERCLIP_OPENCODE_COMMAND = "__paperclip_missing_opencode_command__";
|
||||
const errorSpy = vi.spyOn(console, "error").mockImplementation(() => {});
|
||||
|
||||
const models = await listAdapterModels("opencode_local");
|
||||
expect(models).toEqual(opencodeFallbackModels);
|
||||
|
||||
// Parsing is memoized per raw value: a second call must not re-log.
|
||||
const callsAfterFirst = errorSpy.mock.calls.length;
|
||||
expect(callsAfterFirst).toBeGreaterThan(0);
|
||||
await listAdapterModels("opencode_local");
|
||||
expect(errorSpy.mock.calls.length).toBe(callsAfterFirst);
|
||||
});
|
||||
|
||||
it("ignores declared models for adapters not in the map", async () => {
|
||||
process.env.PAPERCLIP_ADAPTER_MODELS = JSON.stringify({
|
||||
opencode_local: [{ id: "model-a" }],
|
||||
});
|
||||
const models = await listAdapterModels("codex_local");
|
||||
expect(models).toEqual(codexFallbackModels);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { agents, companies, createDb, environmentLeases, environments, heartbeatRuns } from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
@@ -222,6 +222,134 @@ describeEmbeddedPostgres("environmentService leases", () => {
|
||||
expect(rows[0]?.status).toBe("active");
|
||||
});
|
||||
|
||||
it("ensures, refreshes, and finds a managed Kubernetes sandbox environment", async () => {
|
||||
const companyId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Acme",
|
||||
status: "active",
|
||||
createdAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
});
|
||||
|
||||
// No managed k8s env yet.
|
||||
expect(await svc.findKubernetesEnvironment(companyId)).toBeNull();
|
||||
|
||||
const created = await svc.ensureKubernetesEnvironment(companyId, {
|
||||
backend: "job",
|
||||
inCluster: true,
|
||||
runtimeClassName: "gvisor",
|
||||
egressMode: "cilium",
|
||||
egressAllowFqdns: ["api.anthropic.com"],
|
||||
});
|
||||
|
||||
expect(created.driver).toBe("sandbox");
|
||||
expect(created.config.provider).toBe("kubernetes");
|
||||
expect(created.config.backend).toBe("job");
|
||||
expect(created.config.runtimeClassName).toBe("gvisor");
|
||||
expect(created.metadata?.managedKubernetesSandbox).toBe(true);
|
||||
|
||||
// Idempotent: second call refreshes config in place, no new row.
|
||||
const refreshed = await svc.ensureKubernetesEnvironment(companyId, {
|
||||
backend: "job",
|
||||
inCluster: true,
|
||||
egressMode: "cilium",
|
||||
egressAllowFqdns: ["api.anthropic.com", "api.openai.com"],
|
||||
});
|
||||
expect(refreshed.id).toBe(created.id);
|
||||
expect(refreshed.config.egressAllowFqdns).toEqual([
|
||||
"api.anthropic.com",
|
||||
"api.openai.com",
|
||||
]);
|
||||
|
||||
const found = await svc.findKubernetesEnvironment(companyId);
|
||||
expect(found?.id).toBe(created.id);
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(environments)
|
||||
.where(eq(environments.companyId, companyId));
|
||||
expect(rows.filter((row) => row.driver === "sandbox")).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("deduplicates concurrent managed Kubernetes environment creation", async () => {
|
||||
const companyId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Acme",
|
||||
status: "active",
|
||||
createdAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
});
|
||||
|
||||
// No partial unique index covers sandbox drivers yet, so dedup is
|
||||
// post-insert convergence (prefer the oldest row, delete the loser).
|
||||
const results = await Promise.all(
|
||||
Array.from({ length: 8 }, () =>
|
||||
svc.ensureKubernetesEnvironment(companyId, { inCluster: true, backend: "job" }),
|
||||
),
|
||||
);
|
||||
|
||||
expect(new Set(results.map((environment) => environment.id)).size).toBe(1);
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(environments)
|
||||
.where(and(eq(environments.companyId, companyId), eq(environments.driver, "sandbox")));
|
||||
expect(rows).toHaveLength(1);
|
||||
expect((rows[0]?.metadata as Record<string, unknown>)?.managedKubernetesSandbox).toBe(true);
|
||||
});
|
||||
|
||||
it("does not treat a non-kubernetes sandbox environment as the managed k8s env", async () => {
|
||||
const companyId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Acme",
|
||||
status: "active",
|
||||
createdAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
});
|
||||
await svc.create(companyId, {
|
||||
name: "Fake Sandbox",
|
||||
driver: "sandbox",
|
||||
config: { provider: "fake", image: "busybox", reuseLease: false },
|
||||
});
|
||||
|
||||
expect(await svc.findKubernetesEnvironment(companyId)).toBeNull();
|
||||
});
|
||||
|
||||
it("ignores a config.provider=kubernetes sandbox env without the managed marker", async () => {
|
||||
const companyId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Acme",
|
||||
status: "active",
|
||||
createdAt: new Date(),
|
||||
updatedAt: new Date(),
|
||||
});
|
||||
|
||||
// A tenant-created sandbox env with config.provider "kubernetes" but WITHOUT
|
||||
// the managed metadata marker must NOT be treated as the managed k8s env,
|
||||
// otherwise it would bypass the operator gVisor runtimeClass / Cilium egress.
|
||||
await svc.create(companyId, {
|
||||
name: "Tenant K8s Sandbox",
|
||||
driver: "sandbox",
|
||||
config: { provider: "kubernetes", reuseLease: false },
|
||||
});
|
||||
|
||||
expect(await svc.findKubernetesEnvironment(companyId)).toBeNull();
|
||||
|
||||
// The managed env (created via ensureKubernetesEnvironment) carries the
|
||||
// marker and is the only one found.
|
||||
const managed = await svc.ensureKubernetesEnvironment(companyId, {
|
||||
backend: "job",
|
||||
inCluster: true,
|
||||
runtimeClassName: "gvisor",
|
||||
});
|
||||
const found = await svc.findKubernetesEnvironment(companyId);
|
||||
expect(found?.id).toBe(managed.id);
|
||||
});
|
||||
|
||||
it("allows multiple SSH environments for the same company", async () => {
|
||||
const companyId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
|
||||
@@ -210,6 +210,11 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => {
|
||||
config: { template: "base" },
|
||||
runId: run!.id,
|
||||
workspaceMode: "shared_workspace",
|
||||
// Pins the HEARTBEAT-path lease call forwarding the AGENT's adapter type
|
||||
// (per-run adapter / mixed-harness envs). environment-runtime.ts has two
|
||||
// drivers calling environmentAcquireLease; regressions here previously
|
||||
// shipped by editing only the non-heartbeat one.
|
||||
adapterType: "codex_local",
|
||||
});
|
||||
await vi.waitFor(() => {
|
||||
expect(workerManager.call).toHaveBeenCalledWith(pluginId, "environmentReleaseLease", {
|
||||
@@ -427,6 +432,7 @@ describeEmbeddedPostgres("heartbeat plugin environments", () => {
|
||||
config: { template: "new" },
|
||||
runId: run!.id,
|
||||
workspaceMode: "shared_workspace",
|
||||
adapterType: "codex_local",
|
||||
});
|
||||
}, 15_000);
|
||||
});
|
||||
|
||||
@@ -168,6 +168,26 @@ describe("buildPluginWorkerEnv", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("passes in-cluster Kubernetes service-discovery vars to environment driver plugins", () => {
|
||||
const env = buildPluginWorkerEnv({
|
||||
manifest: { capabilities: ["environment.drivers.register"] },
|
||||
instanceInfo,
|
||||
processEnv: {
|
||||
KUBERNETES_SERVICE_HOST: "10.0.0.1",
|
||||
KUBERNETES_SERVICE_PORT: "443",
|
||||
KUBERNETES_SERVICE_PORT_HTTPS: " ",
|
||||
AWS_SECRET_ACCESS_KEY: "aws-secret",
|
||||
},
|
||||
});
|
||||
|
||||
expect(env).toEqual({
|
||||
PAPERCLIP_DEPLOYMENT_MODE: "authenticated",
|
||||
PAPERCLIP_DEPLOYMENT_EXPOSURE: "public",
|
||||
KUBERNETES_SERVICE_HOST: "10.0.0.1",
|
||||
KUBERNETES_SERVICE_PORT: "443",
|
||||
});
|
||||
});
|
||||
|
||||
it("does not pass provider keys to non-environment plugins", () => {
|
||||
const env = buildPluginWorkerEnv({
|
||||
manifest: { capabilities: ["ui.slots.register"] },
|
||||
|
||||
@@ -143,6 +143,7 @@ vi.mock("../services/index.js", () => ({
|
||||
humanGrantsInserted: 0,
|
||||
})),
|
||||
feedbackService: feedbackServiceFactoryMock,
|
||||
bootstrapExecutionPolicyFromEnv: vi.fn(async () => null),
|
||||
heartbeatService: vi.fn(() => ({
|
||||
reapOrphanedRuns: vi.fn(async () => undefined),
|
||||
promoteDueScheduledRetries: vi.fn(async () => ({ promoted: 0, runIds: [] })),
|
||||
|
||||
Reference in New Issue
Block a user