4ad94d0bde
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The execution subsystem runs those agents in environments (local, ssh, sandbox), and sandbox-provider plugins let an environment materialize per-run sandboxes > - Stage 1 (#5790) contributed a first-party Kubernetes sandbox-provider plugin, but the server core has no way to adopt it operationally: no per-run adapter selection, no way to force an instance onto sandboxed execution, no declarative adapter/model configuration, and the plugin must be installed by hand > - Without this, a multi-tenant or security-conscious deployment cannot guarantee that agent runs never execute on the host, and a single environment cannot serve agents with different harnesses > - This pull request adds the server + SDK integration: per-run adapterType on the lease protocol, an env-gated forced-Kubernetes execution policy with provisioning and a per-run allowlist guard, a declarative adapter registry and model list, in-cluster env passthrough for sandbox plugin workers, fail-safe auto-install of the bundled plugin, and the matching UI affordance > - The benefit is that sandbox-provider plugins become fully usable for Kubernetes execution: operators configure everything via environment variables and GitOps, while self-hosters who set none of the variables see exactly the behavior they have today ## Linked Issues or Issue Description Refs #5790 (stage 1 of 3: the Kubernetes sandbox-provider plugin package). No existing issue. Feature description: the server core lacks the integration seams to operate a sandbox-provider plugin as the mandatory execution path of an instance. This PR is stage 2 of 3 of the staged Kubernetes contribution; stage 3 will contribute the agent runtime images and their build pipeline. ## What Changed One line per piece: - `packages/plugins/sdk/protocol.ts`: optional `adapterType` on `PluginEnvironmentAcquireLeaseParams` so a provider can select the runtime image per run; existing providers simply ignore it - `server/services/environment-runtime.ts` + `environment-run-orchestrator.ts`: thread the agent's adapter type into both lease-acquiring drivers, including the heartbeat path (the two call sites have historically drifted, hence the pinned test) - `server/services/environments.ts`: `ensureKubernetesEnvironment` / `findKubernetesEnvironment`, an idempotent managed Kubernetes environment per company, identified by a metadata marker and refreshed (not recreated) on config change; `timeoutMs` rides on the config for slow cold-start leases - `server/services/execution-allowlist.ts`: pure (driver, provider, policy) -> allow/deny guard; `executionMode=kubernetes` only allows the kubernetes sandbox provider - `server/services/execution-policy-bootstrap.ts` + startup hook in `server/index.ts`: parse `PAPERCLIP_EXECUTION_MODE` / `PAPERCLIP_K8S_*`, persist `executionMode` into instance general settings, and provision the managed environment for every company; fails loud on misconfiguration - `server/services/heartbeat.ts`: when the policy forces Kubernetes, pin run selection to the managed environment (also overriding any persisted workspace environment id), refuse to fall back to local, and re-check the actually acquired environment against the allowlist as defense in depth - `server/services/adapter-registry-bootstrap.ts` + shared `AdapterRegistryEntry` type/validator: declarative `PAPERCLIP_ADAPTERS` registry (inline JSON or file) that reconciles adapter availability at startup and rides on the Kubernetes environment config - `server/services/adapter-models-env.ts` + `adapters/registry.ts`: `PAPERCLIP_ADAPTER_MODELS` lets an operator declare picker model lists the server cannot CLI-discover - `server/services/plugin-loader.ts`: pass `KUBERNETES_SERVICE_HOST/PORT(_HTTPS)` through to plugin workers that register environment drivers, so in-cluster API clients can be constructed; all other host env stays stripped - `server/app.ts`: fail-safe auto-install of the bundled kubernetes plugin at boot; no-ops when the bundle is absent and never blocks startup on error - `packages/shared` types/validators: `InstanceExecutionMode` on general settings (optional, strict schema) - `ui/lib/forced-kubernetes-environment.ts` + `AgentConfigForm`: when the policy is active, show a read-only Kubernetes environment instead of the environment picker and default new agents onto the managed environment - Tests for every new module plus the adapterType pin in `heartbeat-plugin-environment` and the managed-environment lifecycle in `environment-service` Everything is gated: with `PAPERCLIP_EXECUTION_MODE`, `PAPERCLIP_ADAPTERS`, and `PAPERCLIP_ADAPTER_MODELS` unset (and no bundled plugin present), every code path reduces to current behavior. The per-run `adapterType` is an optional SDK parameter that existing providers ignore. ## Verification - `cd server && npx tsc --noEmit`: clean (0 errors); `ui` typecheck also clean - Targeted suites all green (11 files, 90 tests): `npx vitest run server/src/__tests__/heartbeat-plugin-environment.test.ts server/src/__tests__/environment-service.test.ts server/src/__tests__/environment-runtime.test.ts server/src/__tests__/environment-run-orchestrator.test.ts server/src/__tests__/plugin-database.test.ts server/src/services/execution-policy-bootstrap.test.ts server/src/services/execution-allowlist.test.ts server/src/services/adapter-registry-bootstrap.test.ts server/src/services/adapter-registry-bootstrap.reconcile.test.ts server/src/services/adapter-models-env.test.ts packages/shared/src/validators/adapter-registry.test.ts` - `npx vitest run ui/src/components/AgentConfigForm.test.ts`: green (6 tests) - Full `npx vitest run server/src/__tests__`: 2323 passed, 1 skipped; the only failures (heartbeat-process-recovery pid-retry, workspace-runtime symbolic-ref/git tests) reproduce identically on pristine `master` in the same environment, so they are machine-environment issues unrelated to this change; `server-startup-feedback-export` needed its `services/index.js` mock extended with the new export and is green - This integration has been running in production on a hosted multi-tenant deployment, where it executes agent runs across five different harnesses through the stage 1 plugin ## Risks - Low for existing deployments: every behavior is env-gated and the defaults preserve current semantics; the auto-install block is wrapped fail-safe and skips silently when the plugin bundle is absent - `executionMode` is a new optional field on a strict zod schema; absent input normalizes exactly as before - The forced policy intentionally fails runs loudly (rather than falling back to local) when no managed Kubernetes environment exists; this only affects instances that explicitly set `PAPERCLIP_EXECUTION_MODE=kubernetes` ## Model Used Claude Opus 4.8 (claude-opus-4-8, 1M context), extended thinking, agentic tool use via Claude Code. ## UI screenshots The UI change is a new read-only "Execution" section in `AgentConfigForm`, shown only when the instance execution policy forces Kubernetes (`executionMode=kubernetes`); there is no "before" state for it (the section did not exist, and instances without the forced policy render the existing picker unchanged). Captured from the new Storybook stories added in this PR (`Product/Agent Management`): Managed Kubernetes environment present (read-only display, no local/SSH picker):  No managed environment available yet (warning notice, no silent local fallback):  ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
195 lines
7.4 KiB
TypeScript
195 lines
7.4 KiB
TypeScript
/**
|
|
* Cloud execution-policy bootstrap.
|
|
*
|
|
* Lets an operator / gitops deployment force the instance onto the Kubernetes
|
|
* sandbox provider purely via environment variables, with no manual product-API
|
|
* calls. On startup we:
|
|
* 1. Parse `PAPERCLIP_EXECUTION_MODE` (+ `PAPERCLIP_K8S_*`) from the env.
|
|
* 2. Persist `executionMode` into instance general settings (so the per-run
|
|
* heartbeat guard enforces it).
|
|
* 3. Idempotently ensure a configured Kubernetes sandbox environment for every
|
|
* company (mirrors `ensureLocalEnvironment`).
|
|
*
|
|
* The boot hook is *configuration convenience*; the actual security gate is the
|
|
* per-run guard in the heartbeat (see `execution-allowlist.ts`). Even with no
|
|
* boot hook, setting `executionMode=kubernetes` denies local execution.
|
|
*
|
|
* The env-var parsing is a pure function so it is trivially unit-testable.
|
|
*/
|
|
|
|
import type { Db } from "@paperclipai/db";
|
|
import type { InstanceExecutionMode } from "@paperclipai/shared";
|
|
import { logger } from "../middleware/logger.js";
|
|
import { environmentService, type KubernetesEnvironmentConfigInput } from "./environments.js";
|
|
import { instanceSettingsService } from "./instance-settings.js";
|
|
import { parseAdapterRegistryEnv } from "./adapter-registry-bootstrap.js";
|
|
|
|
export type ExecutionPolicyBootstrapEnv = Record<string, string | undefined>;
|
|
|
|
export interface ExecutionPolicyBootstrap {
|
|
executionMode: Extract<InstanceExecutionMode, "kubernetes">;
|
|
kubernetesConfig: KubernetesEnvironmentConfigInput;
|
|
}
|
|
|
|
function parseBool(value: string | undefined): boolean | undefined {
|
|
if (value === undefined) return undefined;
|
|
const v = value.trim().toLowerCase();
|
|
if (v === "true" || v === "1" || v === "yes") return true;
|
|
if (v === "false" || v === "0" || v === "no") return false;
|
|
return undefined;
|
|
}
|
|
|
|
function parsePositiveIntMs(value: string | undefined): number | undefined {
|
|
if (value === undefined) return undefined;
|
|
const trimmed = value.trim();
|
|
if (trimmed.length === 0) return undefined;
|
|
const parsed = Number(trimmed);
|
|
if (!Number.isFinite(parsed) || !Number.isInteger(parsed) || parsed <= 0) {
|
|
throw new Error(
|
|
`PAPERCLIP_K8S_RPC_TIMEOUT_MS must be a positive integer of milliseconds (got "${value}").`,
|
|
);
|
|
}
|
|
return parsed;
|
|
}
|
|
|
|
function parseList(value: string | undefined): string[] | undefined {
|
|
if (value === undefined) return undefined;
|
|
const items = value
|
|
.split(",")
|
|
.map((s) => s.trim())
|
|
.filter((s) => s.length > 0);
|
|
return items.length > 0 ? items : undefined;
|
|
}
|
|
|
|
/**
|
|
* Parse the forced-execution-mode env config. Returns null when execution is
|
|
* unrestricted (no env, or `PAPERCLIP_EXECUTION_MODE=any`). Throws on an
|
|
* unrecognized mode so a misconfigured deployment fails loudly instead of
|
|
* silently allowing local execution.
|
|
*/
|
|
export function parseExecutionPolicyBootstrapEnv(
|
|
env: ExecutionPolicyBootstrapEnv,
|
|
): ExecutionPolicyBootstrap | null {
|
|
const raw = env.PAPERCLIP_EXECUTION_MODE?.trim();
|
|
if (!raw || raw === "any") return null;
|
|
if (raw !== "kubernetes") {
|
|
throw new Error(
|
|
`PAPERCLIP_EXECUTION_MODE must be "kubernetes" or "any" (got "${raw}").`,
|
|
);
|
|
}
|
|
|
|
const kubernetesConfig: KubernetesEnvironmentConfigInput = {
|
|
// inCluster defaults to false (matches the plugin schema default); an
|
|
// in-cluster cloud deployment sets PAPERCLIP_K8S_IN_CLUSTER=true.
|
|
inCluster: parseBool(env.PAPERCLIP_K8S_IN_CLUSTER) ?? false,
|
|
};
|
|
|
|
const backend = env.PAPERCLIP_K8S_BACKEND?.trim();
|
|
if (backend) {
|
|
if (backend !== "job" && backend !== "sandbox-cr") {
|
|
throw new Error(
|
|
`PAPERCLIP_K8S_BACKEND must be "job" or "sandbox-cr" (got "${backend}").`,
|
|
);
|
|
}
|
|
kubernetesConfig.backend = backend;
|
|
}
|
|
|
|
const egressMode = env.PAPERCLIP_K8S_EGRESS_MODE?.trim();
|
|
if (egressMode) {
|
|
if (egressMode !== "cilium" && egressMode !== "standard") {
|
|
throw new Error(
|
|
`PAPERCLIP_K8S_EGRESS_MODE must be "cilium" or "standard" (got "${egressMode}").`,
|
|
);
|
|
}
|
|
kubernetesConfig.egressMode = egressMode;
|
|
}
|
|
|
|
const runtimeClassName = env.PAPERCLIP_K8S_RUNTIME_CLASS_NAME?.trim();
|
|
if (runtimeClassName) kubernetesConfig.runtimeClassName = runtimeClassName;
|
|
|
|
const namespacePrefix = env.PAPERCLIP_K8S_NAMESPACE_PREFIX?.trim();
|
|
if (namespacePrefix) kubernetesConfig.namespacePrefix = namespacePrefix;
|
|
|
|
const imageRegistry = env.PAPERCLIP_K8S_IMAGE_REGISTRY?.trim();
|
|
if (imageRegistry) kubernetesConfig.imageRegistry = imageRegistry;
|
|
|
|
const rpcTimeoutMs = parsePositiveIntMs(env.PAPERCLIP_K8S_RPC_TIMEOUT_MS);
|
|
if (rpcTimeoutMs !== undefined) kubernetesConfig.timeoutMs = rpcTimeoutMs;
|
|
|
|
const adapterType = env.PAPERCLIP_K8S_ADAPTER_TYPE?.trim();
|
|
if (adapterType) kubernetesConfig.adapterType = adapterType;
|
|
|
|
const egressAllowFqdns = parseList(env.PAPERCLIP_K8S_EGRESS_ALLOW_FQDNS);
|
|
if (egressAllowFqdns) kubernetesConfig.egressAllowFqdns = egressAllowFqdns;
|
|
|
|
const egressAllowCidrs = parseList(env.PAPERCLIP_K8S_EGRESS_ALLOW_CIDRS);
|
|
if (egressAllowCidrs) kubernetesConfig.egressAllowCidrs = egressAllowCidrs;
|
|
|
|
const adapters = parseAdapterRegistryEnv(env);
|
|
if (adapters) kubernetesConfig.adapters = adapters;
|
|
|
|
return { executionMode: "kubernetes", kubernetesConfig };
|
|
}
|
|
|
|
/**
|
|
* Apply the parsed bootstrap to the database: persist `executionMode` into
|
|
* instance settings and ensure a configured Kubernetes environment for every
|
|
* company. Idempotent; safe to call on every boot.
|
|
*/
|
|
export async function applyExecutionPolicyBootstrap(
|
|
db: Db,
|
|
bootstrap: ExecutionPolicyBootstrap,
|
|
): Promise<{ executionMode: InstanceExecutionMode; companiesConfigured: number }> {
|
|
const instanceSettings = instanceSettingsService(db);
|
|
const environments = environmentService(db);
|
|
|
|
await instanceSettings.updateGeneral({ executionMode: bootstrap.executionMode });
|
|
|
|
const companyIds = await instanceSettings.listCompanyIds();
|
|
let configured = 0;
|
|
const failedCompanyIds: string[] = [];
|
|
for (const companyId of companyIds) {
|
|
try {
|
|
await environments.ensureKubernetesEnvironment(companyId, bootstrap.kubernetesConfig);
|
|
configured += 1;
|
|
} catch (err) {
|
|
logger.error(
|
|
{ err, companyId },
|
|
"failed to ensure managed Kubernetes environment during execution-policy bootstrap",
|
|
);
|
|
failedCompanyIds.push(companyId);
|
|
}
|
|
}
|
|
|
|
logger.info(
|
|
{
|
|
executionMode: bootstrap.executionMode,
|
|
companiesConfigured: configured,
|
|
backend: bootstrap.kubernetesConfig.backend,
|
|
runtimeClassName: bootstrap.kubernetesConfig.runtimeClassName,
|
|
egressMode: bootstrap.kubernetesConfig.egressMode,
|
|
},
|
|
"applied forced Kubernetes execution policy",
|
|
);
|
|
|
|
if (failedCompanyIds.length > 0) {
|
|
throw new Error(
|
|
`execution-policy bootstrap: ${failedCompanyIds.length} of ${companyIds.length} companies failed to get a managed Kubernetes environment under executionMode=${bootstrap.executionMode}; refusing to start (companies: ${failedCompanyIds.join(", ")})`,
|
|
);
|
|
}
|
|
|
|
return { executionMode: bootstrap.executionMode, companiesConfigured: configured };
|
|
}
|
|
|
|
/**
|
|
* Convenience: parse + apply from a raw env map. Returns null when unrestricted.
|
|
*/
|
|
export async function bootstrapExecutionPolicyFromEnv(
|
|
db: Db,
|
|
env: ExecutionPolicyBootstrapEnv = process.env,
|
|
): Promise<{ executionMode: InstanceExecutionMode; companiesConfigured: number } | null> {
|
|
const bootstrap = parseExecutionPolicyBootstrapEnv(env);
|
|
if (!bootstrap) return null;
|
|
return applyExecutionPolicyBootstrap(db, bootstrap);
|
|
}
|