feat(server): kubernetes execution integration for sandbox-provider plugins (stage 2/3) (#7938)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The execution subsystem runs those agents in environments (local, ssh, sandbox), and sandbox-provider plugins let an environment materialize per-run sandboxes > - Stage 1 (#5790) contributed a first-party Kubernetes sandbox-provider plugin, but the server core has no way to adopt it operationally: no per-run adapter selection, no way to force an instance onto sandboxed execution, no declarative adapter/model configuration, and the plugin must be installed by hand > - Without this, a multi-tenant or security-conscious deployment cannot guarantee that agent runs never execute on the host, and a single environment cannot serve agents with different harnesses > - This pull request adds the server + SDK integration: per-run adapterType on the lease protocol, an env-gated forced-Kubernetes execution policy with provisioning and a per-run allowlist guard, a declarative adapter registry and model list, in-cluster env passthrough for sandbox plugin workers, fail-safe auto-install of the bundled plugin, and the matching UI affordance > - The benefit is that sandbox-provider plugins become fully usable for Kubernetes execution: operators configure everything via environment variables and GitOps, while self-hosters who set none of the variables see exactly the behavior they have today ## Linked Issues or Issue Description Refs #5790 (stage 1 of 3: the Kubernetes sandbox-provider plugin package). No existing issue. Feature description: the server core lacks the integration seams to operate a sandbox-provider plugin as the mandatory execution path of an instance. This PR is stage 2 of 3 of the staged Kubernetes contribution; stage 3 will contribute the agent runtime images and their build pipeline. ## What Changed One line per piece: - `packages/plugins/sdk/protocol.ts`: optional `adapterType` on `PluginEnvironmentAcquireLeaseParams` so a provider can select the runtime image per run; existing providers simply ignore it - `server/services/environment-runtime.ts` + `environment-run-orchestrator.ts`: thread the agent's adapter type into both lease-acquiring drivers, including the heartbeat path (the two call sites have historically drifted, hence the pinned test) - `server/services/environments.ts`: `ensureKubernetesEnvironment` / `findKubernetesEnvironment`, an idempotent managed Kubernetes environment per company, identified by a metadata marker and refreshed (not recreated) on config change; `timeoutMs` rides on the config for slow cold-start leases - `server/services/execution-allowlist.ts`: pure (driver, provider, policy) -> allow/deny guard; `executionMode=kubernetes` only allows the kubernetes sandbox provider - `server/services/execution-policy-bootstrap.ts` + startup hook in `server/index.ts`: parse `PAPERCLIP_EXECUTION_MODE` / `PAPERCLIP_K8S_*`, persist `executionMode` into instance general settings, and provision the managed environment for every company; fails loud on misconfiguration - `server/services/heartbeat.ts`: when the policy forces Kubernetes, pin run selection to the managed environment (also overriding any persisted workspace environment id), refuse to fall back to local, and re-check the actually acquired environment against the allowlist as defense in depth - `server/services/adapter-registry-bootstrap.ts` + shared `AdapterRegistryEntry` type/validator: declarative `PAPERCLIP_ADAPTERS` registry (inline JSON or file) that reconciles adapter availability at startup and rides on the Kubernetes environment config - `server/services/adapter-models-env.ts` + `adapters/registry.ts`: `PAPERCLIP_ADAPTER_MODELS` lets an operator declare picker model lists the server cannot CLI-discover - `server/services/plugin-loader.ts`: pass `KUBERNETES_SERVICE_HOST/PORT(_HTTPS)` through to plugin workers that register environment drivers, so in-cluster API clients can be constructed; all other host env stays stripped - `server/app.ts`: fail-safe auto-install of the bundled kubernetes plugin at boot; no-ops when the bundle is absent and never blocks startup on error - `packages/shared` types/validators: `InstanceExecutionMode` on general settings (optional, strict schema) - `ui/lib/forced-kubernetes-environment.ts` + `AgentConfigForm`: when the policy is active, show a read-only Kubernetes environment instead of the environment picker and default new agents onto the managed environment - Tests for every new module plus the adapterType pin in `heartbeat-plugin-environment` and the managed-environment lifecycle in `environment-service` Everything is gated: with `PAPERCLIP_EXECUTION_MODE`, `PAPERCLIP_ADAPTERS`, and `PAPERCLIP_ADAPTER_MODELS` unset (and no bundled plugin present), every code path reduces to current behavior. The per-run `adapterType` is an optional SDK parameter that existing providers ignore. ## Verification - `cd server && npx tsc --noEmit`: clean (0 errors); `ui` typecheck also clean - Targeted suites all green (11 files, 90 tests): `npx vitest run server/src/__tests__/heartbeat-plugin-environment.test.ts server/src/__tests__/environment-service.test.ts server/src/__tests__/environment-runtime.test.ts server/src/__tests__/environment-run-orchestrator.test.ts server/src/__tests__/plugin-database.test.ts server/src/services/execution-policy-bootstrap.test.ts server/src/services/execution-allowlist.test.ts server/src/services/adapter-registry-bootstrap.test.ts server/src/services/adapter-registry-bootstrap.reconcile.test.ts server/src/services/adapter-models-env.test.ts packages/shared/src/validators/adapter-registry.test.ts` - `npx vitest run ui/src/components/AgentConfigForm.test.ts`: green (6 tests) - Full `npx vitest run server/src/__tests__`: 2323 passed, 1 skipped; the only failures (heartbeat-process-recovery pid-retry, workspace-runtime symbolic-ref/git tests) reproduce identically on pristine `master` in the same environment, so they are machine-environment issues unrelated to this change; `server-startup-feedback-export` needed its `services/index.js` mock extended with the new export and is green - This integration has been running in production on a hosted multi-tenant deployment, where it executes agent runs across five different harnesses through the stage 1 plugin ## Risks - Low for existing deployments: every behavior is env-gated and the defaults preserve current semantics; the auto-install block is wrapped fail-safe and skips silently when the plugin bundle is absent - `executionMode` is a new optional field on a strict zod schema; absent input normalizes exactly as before - The forced policy intentionally fails runs loudly (rather than falling back to local) when no managed Kubernetes environment exists; this only affects instances that explicitly set `PAPERCLIP_EXECUTION_MODE=kubernetes` ## Model Used Claude Opus 4.8 (claude-opus-4-8, 1M context), extended thinking, agentic tool use via Claude Code. ## UI screenshots The UI change is a new read-only "Execution" section in `AgentConfigForm`, shown only when the instance execution policy forces Kubernetes (`executionMode=kubernetes`); there is no "before" state for it (the section did not exist, and instances without the forced policy render the existing picker unchanged). Captured from the new Storybook stories added in this PR (`Product/Agent Management`): Managed Kubernetes environment present (read-only display, no local/SSH picker):  No managed environment available yet (warning notice, no silent local fallback):  ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jannes Stubbemann
committed by
GitHub
GitHub
parent
05ab45225a
commit
4ad94d0bde
@@ -1,5 +1,7 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import type { Environment } from "@paperclipai/shared";
|
||||
import { supportsAdapterModelRefresh } from "./AgentConfigForm";
|
||||
import { resolveForcedKubernetesEnvironment } from "../lib/forced-kubernetes-environment";
|
||||
|
||||
describe("supportsAdapterModelRefresh", () => {
|
||||
it("enables the model refresh action for Claude, Codex, and ACPX adapters", () => {
|
||||
@@ -13,3 +15,58 @@ describe("supportsAdapterModelRefresh", () => {
|
||||
expect(supportsAdapterModelRefresh("process")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
function makeEnvironment(overrides: Partial<Environment>): Environment {
|
||||
return {
|
||||
id: "env-1",
|
||||
companyId: "co-1",
|
||||
name: "Env",
|
||||
description: null,
|
||||
driver: "local",
|
||||
status: "active",
|
||||
config: {},
|
||||
metadata: null,
|
||||
createdAt: new Date(0),
|
||||
updatedAt: new Date(0),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
const localEnv = makeEnvironment({ id: "local-1", name: "Local", driver: "local" });
|
||||
const k8sEnv = makeEnvironment({
|
||||
id: "k8s-1",
|
||||
name: "Managed K8s",
|
||||
driver: "sandbox",
|
||||
config: { provider: "kubernetes" },
|
||||
});
|
||||
|
||||
describe("resolveForcedKubernetesEnvironment", () => {
|
||||
it("does not force when executionMode is 'any' (full picker / unchanged)", () => {
|
||||
const result = resolveForcedKubernetesEnvironment("any", [localEnv, k8sEnv]);
|
||||
expect(result.forced).toBe(false);
|
||||
expect(result.kubernetesEnvironment).toBeNull();
|
||||
});
|
||||
|
||||
it("does not force when executionMode is absent (self-hoster default)", () => {
|
||||
const result = resolveForcedKubernetesEnvironment(undefined, [localEnv, k8sEnv]);
|
||||
expect(result.forced).toBe(false);
|
||||
expect(result.kubernetesEnvironment).toBeNull();
|
||||
});
|
||||
|
||||
it("forces and selects the Kubernetes sandbox when executionMode is 'kubernetes'", () => {
|
||||
const result = resolveForcedKubernetesEnvironment("kubernetes", [localEnv, k8sEnv]);
|
||||
expect(result.forced).toBe(true);
|
||||
expect(result.kubernetesEnvironment?.id).toBe("k8s-1");
|
||||
});
|
||||
|
||||
it("forces but reports no environment when none is the Kubernetes sandbox", () => {
|
||||
const fakeSandbox = makeEnvironment({
|
||||
id: "fake-1",
|
||||
driver: "sandbox",
|
||||
config: { provider: "fake" },
|
||||
});
|
||||
const result = resolveForcedKubernetesEnvironment("kubernetes", [localEnv, fakeSandbox]);
|
||||
expect(result.forced).toBe(true);
|
||||
expect(result.kubernetesEnvironment).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -55,6 +55,7 @@ import { useDisabledAdaptersSync } from "../adapters/use-disabled-adapters";
|
||||
import { buildAgentUpdatePatch, type AgentConfigOverlay } from "../lib/agent-config-patch";
|
||||
import { useAdapterCapabilities } from "../adapters/use-adapter-capabilities";
|
||||
import { filterAcpxModelsByAgent } from "../lib/acpx-model-filter";
|
||||
import { resolveForcedKubernetesEnvironment } from "../lib/forced-kubernetes-environment";
|
||||
|
||||
/* ---- Create mode values ---- */
|
||||
|
||||
@@ -223,11 +224,33 @@ export function AgentConfigForm(props: AgentConfigFormProps) {
|
||||
});
|
||||
const environmentsEnabled = experimentalSettings?.enableEnvironments === true;
|
||||
|
||||
// Instance execution policy (general settings). When `executionMode` is
|
||||
// "kubernetes" the instance FORCES all execution onto the managed Kubernetes
|
||||
// sandbox; "any"/absent leaves the full environment/adapter choice intact.
|
||||
// Reuses the same general-settings query the rest of the UI uses.
|
||||
const { data: generalSettings } = useQuery({
|
||||
queryKey: queryKeys.instance.generalSettings,
|
||||
queryFn: () => instanceSettingsApi.getGeneral(),
|
||||
retry: false,
|
||||
});
|
||||
|
||||
const { data: environments = [] } = useQuery<Environment[]>({
|
||||
queryKey: selectedCompanyId ? queryKeys.environments.list(selectedCompanyId) : ["environments", "none"],
|
||||
queryFn: () => environmentsApi.list(selectedCompanyId!),
|
||||
enabled: Boolean(selectedCompanyId) && environmentsEnabled,
|
||||
// Load environments when the picker is enabled OR when execution is forced
|
||||
// onto Kubernetes (so we can resolve and default to the managed K8s env even
|
||||
// when the experimental environments picker is otherwise hidden).
|
||||
enabled:
|
||||
Boolean(selectedCompanyId) &&
|
||||
(environmentsEnabled || generalSettings?.executionMode === "kubernetes"),
|
||||
});
|
||||
|
||||
// Setting-driven: resolve whether the instance forces Kubernetes execution and
|
||||
// which loaded environment is the managed Kubernetes sandbox.
|
||||
const { forced: forcedKubernetes, kubernetesEnvironment } = useMemo(
|
||||
() => resolveForcedKubernetesEnvironment(generalSettings?.executionMode, environments),
|
||||
[generalSettings?.executionMode, environments],
|
||||
);
|
||||
const createSecret = useMutation({
|
||||
mutationFn: (input: { name: string; value: string }) => {
|
||||
if (!selectedCompanyId) throw new Error("Select a company to create secrets");
|
||||
@@ -336,6 +359,17 @@ export function AgentConfigForm(props: AgentConfigFormProps) {
|
||||
() => environments.find((environment) => environment.id === currentDefaultEnvironmentId) ?? null,
|
||||
[currentDefaultEnvironmentId, environments],
|
||||
);
|
||||
|
||||
// When the instance forces Kubernetes execution, new agents must default to the
|
||||
// managed Kubernetes sandbox environment (never the implicit local default).
|
||||
// Only applies in create mode and only once the K8s environment is loaded; if
|
||||
// none is available the UI surfaces a notice instead of silently selecting it.
|
||||
useEffect(() => {
|
||||
if (!isCreate || !set || !forcedKubernetes || !kubernetesEnvironment) return;
|
||||
if (currentDefaultEnvironmentId === kubernetesEnvironment.id) return;
|
||||
set({ defaultEnvironmentId: kubernetesEnvironment.id });
|
||||
}, [isCreate, set, forcedKubernetes, kubernetesEnvironment, currentDefaultEnvironmentId]);
|
||||
|
||||
const runnableEnvironments = useMemo(
|
||||
() => environments.filter((environment) => {
|
||||
if (!supportedEnvironmentDrivers.has(environment.driver)) return false;
|
||||
@@ -782,7 +816,35 @@ export function AgentConfigForm(props: AgentConfigFormProps) {
|
||||
)}
|
||||
|
||||
{/* ---- Execution ---- */}
|
||||
{environmentsEnabled ? (
|
||||
{forcedKubernetes ? (
|
||||
// Instance execution policy forces the managed Kubernetes sandbox
|
||||
// (executionMode=kubernetes): never offer local / non-Kubernetes targets.
|
||||
// Render the environment read-only instead of the selectable picker.
|
||||
<div className={cn(!cards && (isCreate ? "border-t border-border" : "border-b border-border"))}>
|
||||
{cards
|
||||
? <h3 className="text-sm font-medium mb-3">Execution</h3>
|
||||
: <div className="px-4 py-2 text-xs font-medium text-muted-foreground">Execution</div>
|
||||
}
|
||||
<div className={cn(cards ? "border border-border rounded-lg p-4 space-y-3" : "px-4 pb-3 space-y-3")}>
|
||||
<Field
|
||||
label="Default environment"
|
||||
hint="This instance runs all agents in the Kubernetes sandbox. Local execution is disabled."
|
||||
>
|
||||
{kubernetesEnvironment ? (
|
||||
<div className={cn(inputClass, "flex items-center text-muted-foreground")}>
|
||||
{kubernetesEnvironment.name} · Kubernetes sandbox
|
||||
</div>
|
||||
) : (
|
||||
<div className="rounded-md border border-amber-500/25 bg-amber-500/10 px-3 py-2 text-xs text-amber-200">
|
||||
This instance requires the Kubernetes sandbox, but no managed Kubernetes
|
||||
environment is available for this company yet. Configure one before creating
|
||||
agents; execution will not fall back to local.
|
||||
</div>
|
||||
)}
|
||||
</Field>
|
||||
</div>
|
||||
</div>
|
||||
) : environmentsEnabled ? (
|
||||
<div className={cn(!cards && (isCreate ? "border-t border-border" : "border-b border-border"))}>
|
||||
{cards
|
||||
? <h3 className="text-sm font-medium mb-3">Execution</h3>
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
import type { Environment, InstanceExecutionMode } from "@paperclipai/shared";
|
||||
|
||||
/**
|
||||
* Provider key (== plugin driverKey) of the first-party Kubernetes sandbox
|
||||
* provider. Mirrors `KUBERNETES_PROVIDER_KEY` in the server-side execution
|
||||
* allowlist. Kept local to the UI because the allowlist module lives in the
|
||||
* server package and must not be imported by the client bundle.
|
||||
*/
|
||||
export const KUBERNETES_PROVIDER_KEY = "kubernetes" as const;
|
||||
|
||||
/**
|
||||
* True iff the environment is the Kubernetes sandbox provider, i.e. a core
|
||||
* `sandbox` driver whose `config.provider` is "kubernetes". Mirrors the
|
||||
* server-side `isKubernetesSandboxEnvironment` guard so the UI selects exactly
|
||||
* the environment the server forces execution onto.
|
||||
*/
|
||||
export function isKubernetesSandboxEnvironment(environment: Environment): boolean {
|
||||
if (environment.driver !== "sandbox") return false;
|
||||
const provider = environment.config?.provider;
|
||||
return provider === KUBERNETES_PROVIDER_KEY;
|
||||
}
|
||||
|
||||
export interface ForcedKubernetesEnvironment {
|
||||
/**
|
||||
* Whether the instance execution policy forces all execution onto the
|
||||
* Kubernetes sandbox. Driven entirely by the `executionMode` instance general
|
||||
* setting: `"kubernetes"` forces; `"any"`/absent does not. A self-hoster who
|
||||
* keeps the default `"any"` retains the full environment/adapter picker.
|
||||
*/
|
||||
forced: boolean;
|
||||
/**
|
||||
* The company's managed Kubernetes sandbox environment, if one is present in
|
||||
* the loaded environment list. `null` when forced but no such environment is
|
||||
* available yet (the UI should show a clear notice rather than silently
|
||||
* defaulting to local).
|
||||
*/
|
||||
kubernetesEnvironment: Environment | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve whether execution is forced onto Kubernetes and, if so, which loaded
|
||||
* environment is the Kubernetes sandbox. Pure so it can be unit-tested without
|
||||
* rendering.
|
||||
*/
|
||||
export function resolveForcedKubernetesEnvironment(
|
||||
executionMode: InstanceExecutionMode | undefined,
|
||||
environments: readonly Environment[],
|
||||
): ForcedKubernetesEnvironment {
|
||||
const forced = executionMode === "kubernetes";
|
||||
if (!forced) {
|
||||
return { forced: false, kubernetesEnvironment: null };
|
||||
}
|
||||
const kubernetesEnvironment =
|
||||
environments.find((environment) => isKubernetesSandboxEnvironment(environment)) ?? null;
|
||||
return { forced: true, kubernetesEnvironment };
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
type AgentRuntimeState,
|
||||
type CompanySecret,
|
||||
type EnvBinding,
|
||||
type Environment,
|
||||
} from "@paperclipai/shared";
|
||||
import { ActiveAgentsPanel } from "@/components/ActiveAgentsPanel";
|
||||
import { AgentConfigForm, type CreateConfigValues } from "@/components/AgentConfigForm";
|
||||
@@ -774,3 +775,88 @@ export default meta;
|
||||
type Story = StoryObj<typeof meta>;
|
||||
|
||||
export const ManagementMatrix: Story = {};
|
||||
|
||||
/* ---- Forced Kubernetes execution (instance executionMode=kubernetes) ---- */
|
||||
|
||||
const managedKubernetesEnvironment: Environment = {
|
||||
id: "env-k8s-storybook",
|
||||
companyId: COMPANY_ID,
|
||||
name: "Kubernetes Sandbox",
|
||||
description: "Managed Kubernetes sandbox environment for hosted tenant execution.",
|
||||
driver: "sandbox",
|
||||
status: "active",
|
||||
config: {
|
||||
provider: "kubernetes",
|
||||
backend: "job",
|
||||
inCluster: true,
|
||||
runtimeClassName: "gvisor",
|
||||
egressMode: "cilium",
|
||||
},
|
||||
metadata: { managedByPaperclip: true, managedKubernetesSandbox: true },
|
||||
createdAt: recent(2_000),
|
||||
updatedAt: recent(60),
|
||||
};
|
||||
|
||||
function ForcedKubernetesFixtures({
|
||||
environmentFixtures,
|
||||
children,
|
||||
}: {
|
||||
environmentFixtures: Environment[];
|
||||
children: ReactNode;
|
||||
}) {
|
||||
const queryClient = useQueryClient();
|
||||
|
||||
queryClient.setQueryData(queryKeys.agents.list(COMPANY_ID), agentManagementAgents);
|
||||
queryClient.setQueryData(queryKeys.secrets.list(COMPANY_ID), storybookSecrets);
|
||||
queryClient.setQueryData(queryKeys.adapters.all, adapterFixtures);
|
||||
// The instance-level execution policy that forces all agent execution onto
|
||||
// the managed Kubernetes sandbox (PAPERCLIP_EXECUTION_MODE=kubernetes).
|
||||
queryClient.setQueryData(queryKeys.instance.generalSettings, {
|
||||
censorUsernameInLogs: false,
|
||||
executionMode: "kubernetes",
|
||||
});
|
||||
queryClient.setQueryData(queryKeys.environments.list(COMPANY_ID), environmentFixtures);
|
||||
queryClient.setQueryData(queryKeys.agents.adapterModels(COMPANY_ID, "codex_local"), [
|
||||
{ id: "gpt-5.4", label: "GPT-5.4" },
|
||||
{ id: "gpt-5.4-mini", label: "GPT-5.4 Mini" },
|
||||
]);
|
||||
|
||||
return children;
|
||||
}
|
||||
|
||||
function ForcedKubernetesStory({ environmentFixtures }: { environmentFixtures: Environment[] }) {
|
||||
return (
|
||||
<ForcedKubernetesFixtures environmentFixtures={environmentFixtures}>
|
||||
<div className="min-h-screen bg-background text-foreground">
|
||||
<main className="mx-auto max-w-4xl space-y-8 px-6 py-10">
|
||||
<Section
|
||||
eyebrow="AgentConfigForm"
|
||||
title="Execution pinned to the managed Kubernetes sandbox (executionMode=kubernetes)"
|
||||
>
|
||||
<div className="max-w-4xl">
|
||||
<AgentConfigFormStory />
|
||||
</div>
|
||||
</Section>
|
||||
</main>
|
||||
</div>
|
||||
</ForcedKubernetesFixtures>
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Instance execution policy forces Kubernetes and the company has a managed
|
||||
* Kubernetes sandbox environment: the Execution section renders the
|
||||
* environment read-only (no local/SSH picker).
|
||||
*/
|
||||
export const ForcedKubernetesExecution: Story = {
|
||||
render: () => <ForcedKubernetesStory environmentFixtures={[managedKubernetesEnvironment]} />,
|
||||
};
|
||||
|
||||
/**
|
||||
* Instance execution policy forces Kubernetes but no managed environment is
|
||||
* available for the company yet: the Execution section shows the warning
|
||||
* notice instead of silently falling back to local execution.
|
||||
*/
|
||||
export const ForcedKubernetesMissingEnvironment: Story = {
|
||||
render: () => <ForcedKubernetesStory environmentFixtures={[]} />,
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user