[codex] feat(watchdog): add task watchdog control plane (#8339)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task lifecycle and recovery subsystems decide when agent work is still productive, stalled, or ready for review. > - Existing recovery paths can observe stopped or incomplete work, but there was no first-class per-task watchdog model with scoped review permissions. > - Watchdog follow-ups also need strict boundaries so recovery/status-only runs cannot mutate approvals or perform deliverable work. > - This pull request adds the task watchdog data model, API/service layer, scheduler/review flow, adapter wake context, UI configuration surfaces, and docs. > - The branch has been rebased onto current `paperclipai/paperclip` `master`; the watchdog migration is now ordered after master's latest migrations as `0104_issue_watchdogs`. > - The benefit is a more explicit task-review loop that preserves Paperclip's single-assignee and governance invariants while making stalled work easier to route. ## Linked Issues or Issue Description No linked GitHub issue. Paperclip task: [PAP-11275](/PAP/issues/PAP-11275). ## Problem or motivation Task recovery needs a first-class watchdog path that can inspect stopped work and create scoped follow-ups without bypassing normal task ownership. Board/UI users need a way to configure watchdogs on tasks and see watchdog-related live work. Recovery/status-only runs must remain limited to status reporting and must not create approvals, link approvals, or submit approval comments. ## Proposed solution Add a task-watchdog data model, scheduler/classifier, scoped mutation guard, adapter wake context, API/UI configuration surfaces, and documentation so watchdog agents can review stopped task subtrees under explicit boundaries. ## Alternatives considered Reuse the existing recovery-action flow only. That would keep stopped-work detection implicit, make per-task watchdog assignment harder to expose in the UI, and would not provide a durable scoped-review issue for stalled task trees. ## Roadmap alignment This is Paperclip control-plane lifecycle infrastructure for task execution and recovery. I checked `ROADMAP.md`; this PR does not duplicate an existing planned core item. ## What Changed - Added issue watchdog schema, migration, shared contracts, validators, CRUD API, and service support. - Added task watchdog scheduler/classifier behavior, scoped mutation enforcement, adapter wake context, and default watchdog mandate guidance. - Added UI surfaces for configuring watchdogs on new/existing tasks, viewing watchdog activity, and exposing the experimental setting. - Added docs for the user-facing task watchdog workflow and implementation semantics. - Gated new-task watchdog setup behind `enableTaskWatchdogs` and blocked cheap status-only recovery runs from approval mutations. - Rebased onto current `master` and renumbered the idempotent watchdog migration from the branch-local `0102_issue_watchdogs` slot to `0104_issue_watchdogs`. - Addressed Greptile feedback by loading watchdog classifier input with a recursive subtree query and centralizing the watchdog origin-kind constant. - Added and updated focused server/UI tests for watchdog routes, scheduler/classifier behavior, scope boundaries, live task visibility, settings, and new issue dialog behavior. ## Verification - `pnpm vitest run server/src/__tests__/task-watchdogs-scheduler.test.ts server/src/__tests__/task-watchdogs-classifier.test.ts` - `pnpm vitest run server/src/__tests__/approval-routes-idempotency.test.ts server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts` - `pnpm vitest run ui/src/components/NewIssueDialog.test.tsx` - `pnpm --filter @paperclipai/server typecheck` - `git diff --check` - Verified the PR diff does not include `pnpm-lock.yaml` or `.github/workflows`. ## Risks - Medium risk: this introduces a new task lifecycle surface touching DB schema, server routes/services, adapter wake context, and UI task configuration. - Watchdog scheduling behavior depends on the new experimental setting and runtime context checks behaving consistently across local and production agents. - The watchdog migration is idempotent (`IF NOT EXISTS` / duplicate-object guards) so users who tried the previous branch-local migration number should not get duplicate-object failures. - CI and the second Greptile pass are pending after the latest review-fix push. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5-class coding agent in the Paperclip workspace. Exact runtime model id and context window were not exposed to the agent; tool use and local command execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots — N/A per Paperclip task instruction: do not add screenshots/images to this PR unless they are specifically part of the work. - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -158,6 +158,7 @@ import {
|
||||
} from "./recovery/model-profile-hint.js";
|
||||
import { recoveryService } from "./recovery/service.js";
|
||||
import { productivityReviewService } from "./productivity-review.js";
|
||||
import { taskWatchdogService } from "./task-watchdogs.js";
|
||||
import { withAgentStartLock } from "./agent-start-lock.js";
|
||||
import {
|
||||
evaluateAgentInvokability,
|
||||
@@ -2792,6 +2793,7 @@ export async function buildPaperclipWakePayload(input: {
|
||||
? input.contextSnapshot.unresolvedBlockerSummaries
|
||||
: [],
|
||||
executionStage: Object.keys(executionStage).length > 0 ? executionStage : null,
|
||||
taskWatchdog: (input.contextSnapshot.taskWatchdog ?? null) as unknown,
|
||||
continuationSummary: safeContinuationSummary
|
||||
? {
|
||||
key: safeContinuationSummary.key,
|
||||
@@ -3219,6 +3221,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
const budgets = budgetService(db, budgetHooks);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
const productivityReviews = productivityReviewService(db, { enqueueWakeup });
|
||||
const taskWatchdogs = taskWatchdogService(db, { enqueueWakeup });
|
||||
let unsafeTextProjectionPromise: Promise<boolean> | null = null;
|
||||
|
||||
async function releaseEnvironmentLeasesForRun(input: {
|
||||
@@ -7744,6 +7747,10 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
return productivityReviews.reconcileProductivityReviews(opts);
|
||||
}
|
||||
|
||||
async function reconcileTaskWatchdogs(opts?: { companyId?: string | null; runId?: string | null }) {
|
||||
return taskWatchdogs.reconcileTaskWatchdogs(opts);
|
||||
}
|
||||
|
||||
async function buildRunOutputSilence(
|
||||
run: Pick<
|
||||
typeof heartbeatRuns.$inferSelect,
|
||||
@@ -11738,6 +11745,8 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
|
||||
reconcileProductivityReviews,
|
||||
|
||||
reconcileTaskWatchdogs,
|
||||
|
||||
buildRunOutputSilence,
|
||||
|
||||
tickTimers: async (now = new Date()) => {
|
||||
|
||||
@@ -27,6 +27,12 @@ export { issueTreeControlService } from "./issue-tree-control.js";
|
||||
export { issueApprovalService } from "./issue-approvals.js";
|
||||
export { issueReferenceService } from "./issue-references.js";
|
||||
export { issueRecoveryActionService } from "./issue-recovery-actions.js";
|
||||
export { taskWatchdogService } from "./task-watchdogs.js";
|
||||
export {
|
||||
issueIsInTaskWatchdogSubtree,
|
||||
resolveTaskWatchdogMutationScope,
|
||||
taskWatchdogScopeAllowsIssueMutation,
|
||||
} from "./task-watchdog-scope.js";
|
||||
export { goalService } from "./goals.js";
|
||||
export { activityService, type ActivityFilters } from "./activity.js";
|
||||
export { approvalService } from "./approvals.js";
|
||||
|
||||
@@ -49,6 +49,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta
|
||||
enableConferenceRoomChat: parsed.data.enableConferenceRoomChat ?? false,
|
||||
enableIssuePlanDecompositions: parsed.data.enableIssuePlanDecompositions ?? false,
|
||||
enableExperimentalFileViewer: parsed.data.enableExperimentalFileViewer ?? false,
|
||||
enableTaskWatchdogs: parsed.data.enableTaskWatchdogs ?? false,
|
||||
enableCloudSync: parsed.data.enableCloudSync ?? false,
|
||||
autoRestartDevServerWhenIdle: parsed.data.autoRestartDevServerWhenIdle ?? false,
|
||||
enableIssueGraphLivenessAutoRecovery: parsed.data.enableIssueGraphLivenessAutoRecovery ?? false,
|
||||
@@ -64,6 +65,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta
|
||||
enableConferenceRoomChat: false,
|
||||
enableIssuePlanDecompositions: false,
|
||||
enableExperimentalFileViewer: false,
|
||||
enableTaskWatchdogs: false,
|
||||
enableCloudSync: false,
|
||||
autoRestartDevServerWhenIdle: false,
|
||||
enableIssueGraphLivenessAutoRecovery: false,
|
||||
|
||||
@@ -19,6 +19,7 @@ import {
|
||||
issueAttachments,
|
||||
issueInboxArchives,
|
||||
issueLabels,
|
||||
issueWatchdogs,
|
||||
issuePlanDecompositions,
|
||||
issueRecoveryActions,
|
||||
issueRelations,
|
||||
@@ -44,6 +45,7 @@ import type {
|
||||
IssueProductivityReview,
|
||||
IssueProductivityReviewTrigger,
|
||||
IssueRelationIssueSummary,
|
||||
IssueWatchdogSummary,
|
||||
LowTrustBoundary,
|
||||
SuccessfulRunHandoffState,
|
||||
} from "@paperclipai/shared";
|
||||
@@ -76,6 +78,10 @@ import { resolveIssueGoalId, resolveNextIssueGoalId } from "./issue-goal-fallbac
|
||||
import { getRunLogStore } from "./run-log-store.js";
|
||||
import { getDefaultCompanyGoal } from "./goals.js";
|
||||
import { assertAssignableAgent } from "./agent-assignability.js";
|
||||
import {
|
||||
summarizeIssueWatchdog,
|
||||
upsertIssueWatchdogForIssue,
|
||||
} from "./task-watchdogs.js";
|
||||
import {
|
||||
isVerifiedIssueTreeControlInteractionWake,
|
||||
issueTreeControlService,
|
||||
@@ -302,7 +308,11 @@ type IssueScheduledRetryRow = {
|
||||
error?: string | null;
|
||||
errorCode?: string | null;
|
||||
};
|
||||
type IssueWithLabels = IssueRow & { labels: IssueLabelRow[]; labelIds: string[] };
|
||||
type IssueWithLabels = IssueRow & {
|
||||
labels: IssueLabelRow[];
|
||||
labelIds: string[];
|
||||
watchdog?: IssueWatchdogSummary | null;
|
||||
};
|
||||
type IssueWithLabelsAndRun = IssueWithLabels & { activeRun: IssueActiveRunRow | null };
|
||||
type IssueUserCommentStats = {
|
||||
issueId: string;
|
||||
@@ -354,6 +364,8 @@ type IssueCreateInput = Omit<typeof issues.$inferInsert, "companyId"> & {
|
||||
labelIds?: string[];
|
||||
blockedByIssueIds?: string[];
|
||||
inheritExecutionWorkspaceFromIssueId?: string | null;
|
||||
watchdog?: { agentId: string; instructions?: string | null } | null;
|
||||
watchdogActorRunId?: string | null;
|
||||
};
|
||||
type IssueChildCreateInput = IssueCreateInput & {
|
||||
acceptanceCriteria?: string[];
|
||||
@@ -1143,17 +1155,49 @@ async function labelMapForIssues(dbOrTx: any, issueIds: string[]): Promise<Map<s
|
||||
|
||||
async function withIssueLabels(dbOrTx: any, rows: IssueRow[]): Promise<IssueWithLabels[]> {
|
||||
if (rows.length === 0) return [];
|
||||
const labelsByIssueId = await labelMapForIssues(dbOrTx, rows.map((row) => row.id));
|
||||
const issueIds = rows.map((row) => row.id);
|
||||
const [labelsByIssueId, watchdogByIssueId] = await Promise.all([
|
||||
labelMapForIssues(dbOrTx, issueIds),
|
||||
watchdogMapForIssues(dbOrTx, rows),
|
||||
]);
|
||||
return rows.map((row) => {
|
||||
const issueLabels = labelsByIssueId.get(row.id) ?? [];
|
||||
return {
|
||||
...row,
|
||||
labels: issueLabels,
|
||||
labelIds: issueLabels.map((label) => label.id),
|
||||
watchdog: watchdogByIssueId.get(row.id) ?? null,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
async function watchdogMapForIssues(dbOrTx: any, rows: IssueRow[]): Promise<Map<string, IssueWatchdogSummary>> {
|
||||
const map = new Map<string, IssueWatchdogSummary>();
|
||||
if (rows.length === 0) return map;
|
||||
const byCompany = new Map<string, string[]>();
|
||||
for (const row of rows) {
|
||||
const ids = byCompany.get(row.companyId) ?? [];
|
||||
ids.push(row.id);
|
||||
byCompany.set(row.companyId, ids);
|
||||
}
|
||||
for (const [companyId, issueIds] of byCompany.entries()) {
|
||||
for (const issueIdChunk of chunkList([...new Set(issueIds)], ISSUE_LIST_RELATED_QUERY_CHUNK_SIZE)) {
|
||||
const watchdogRows = await dbOrTx
|
||||
.select()
|
||||
.from(issueWatchdogs)
|
||||
.where(and(
|
||||
eq(issueWatchdogs.companyId, companyId),
|
||||
inArray(issueWatchdogs.issueId, issueIdChunk),
|
||||
eq(issueWatchdogs.status, "active"),
|
||||
));
|
||||
for (const row of watchdogRows) {
|
||||
map.set(row.issueId, summarizeIssueWatchdog(row));
|
||||
}
|
||||
}
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
const ACTIVE_RUN_STATUSES = ["queued", "running"];
|
||||
const BLOCKER_ATTENTION_ACTIVE_RUN_STATUSES = ["queued", "running"];
|
||||
const BLOCKER_ATTENTION_ACTIVE_WAKE_STATUSES = ["queued", "deferred_issue_execution"];
|
||||
@@ -4857,6 +4901,8 @@ export function issueService(db: Db) {
|
||||
labelIds: inputLabelIds,
|
||||
blockedByIssueIds,
|
||||
inheritExecutionWorkspaceFromIssueId,
|
||||
watchdog,
|
||||
watchdogActorRunId,
|
||||
...issueData
|
||||
} = data;
|
||||
const isolatedWorkspacesEnabled = (await instanceSettings.getExperimental()).enableIsolatedWorkspaces;
|
||||
@@ -5073,6 +5119,17 @@ export function issueService(db: Db) {
|
||||
);
|
||||
|
||||
const [issue] = await tx.insert(issues).values(values).returning();
|
||||
if (watchdog) {
|
||||
await upsertIssueWatchdogForIssue(tx, companyId, issue.id, {
|
||||
agentId: watchdog.agentId,
|
||||
instructions: watchdog.instructions,
|
||||
actor: {
|
||||
agentId: issueData.createdByAgentId ?? null,
|
||||
userId: issueData.createdByUserId ?? null,
|
||||
runId: watchdogActorRunId ?? null,
|
||||
},
|
||||
});
|
||||
}
|
||||
if (inputLabelIds) {
|
||||
await syncIssueLabels(issue.id, companyId, inputLabelIds, tx);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { heartbeatRuns, issues, issueWatchdogs } from "@paperclipai/db";
|
||||
|
||||
const MAX_WATCHDOG_SCOPE_ANCESTRY_DEPTH = 100;
|
||||
export const TASK_WATCHDOG_ORIGIN_KIND = "task_watchdog";
|
||||
|
||||
type AgentRunActor = {
|
||||
type: string;
|
||||
agentId?: string | null;
|
||||
companyId?: string | null;
|
||||
runId?: string | null;
|
||||
};
|
||||
|
||||
type IssueScopeTarget = {
|
||||
id: string;
|
||||
companyId: string;
|
||||
parentId?: string | null;
|
||||
};
|
||||
|
||||
export type TaskWatchdogMutationScope =
|
||||
| { kind: "none" }
|
||||
| { kind: "invalid"; detail: string }
|
||||
| {
|
||||
kind: "watchdog";
|
||||
watchdogId: string;
|
||||
companyId: string;
|
||||
watchedIssueId: string;
|
||||
watchdogIssueId: string | null;
|
||||
stopFingerprint: string | null;
|
||||
};
|
||||
|
||||
function isPlainRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function readString(value: unknown): string | null {
|
||||
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
|
||||
}
|
||||
|
||||
function readTaskWatchdogContext(contextSnapshot: unknown) {
|
||||
const context = isPlainRecord(contextSnapshot) ? contextSnapshot : null;
|
||||
const taskWatchdog = isPlainRecord(context?.taskWatchdog) ? context.taskWatchdog : null;
|
||||
if (!taskWatchdog && context?.taskWatchdog !== true) return null;
|
||||
return {
|
||||
watchedIssueId: readString(taskWatchdog?.watchedIssueId) ?? readString(context?.watchedIssueId),
|
||||
stopFingerprint: readString(taskWatchdog?.stopFingerprint) ?? readString(context?.stopFingerprint),
|
||||
};
|
||||
}
|
||||
|
||||
export async function resolveTaskWatchdogMutationScope(
|
||||
db: Db,
|
||||
actor: AgentRunActor,
|
||||
): Promise<TaskWatchdogMutationScope> {
|
||||
if (actor.type !== "agent") return { kind: "none" };
|
||||
const agentId = readString(actor.agentId);
|
||||
const runId = readString(actor.runId);
|
||||
const actorCompanyId = readString(actor.companyId);
|
||||
if (!agentId || !runId) return { kind: "none" };
|
||||
|
||||
const run = await db
|
||||
.select({
|
||||
id: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
agentId: heartbeatRuns.agentId,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, runId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
if (!run) return { kind: "none" };
|
||||
const taskWatchdog = readTaskWatchdogContext(run.contextSnapshot);
|
||||
if (!taskWatchdog) return { kind: "none" };
|
||||
if (run.agentId !== agentId || (actorCompanyId && run.companyId !== actorCompanyId)) {
|
||||
return {
|
||||
kind: "invalid",
|
||||
detail: "Task-watchdog run context does not belong to this agent.",
|
||||
};
|
||||
}
|
||||
|
||||
if (!taskWatchdog.watchedIssueId) {
|
||||
return {
|
||||
kind: "invalid",
|
||||
detail: "Task-watchdog run context is missing a persisted watched issue id.",
|
||||
};
|
||||
}
|
||||
|
||||
const watchdog = await db
|
||||
.select({
|
||||
id: issueWatchdogs.id,
|
||||
companyId: issueWatchdogs.companyId,
|
||||
issueId: issueWatchdogs.issueId,
|
||||
watchdogAgentId: issueWatchdogs.watchdogAgentId,
|
||||
watchdogIssueId: issueWatchdogs.watchdogIssueId,
|
||||
status: issueWatchdogs.status,
|
||||
})
|
||||
.from(issueWatchdogs)
|
||||
.where(and(
|
||||
eq(issueWatchdogs.companyId, run.companyId),
|
||||
eq(issueWatchdogs.issueId, taskWatchdog.watchedIssueId),
|
||||
eq(issueWatchdogs.watchdogAgentId, agentId),
|
||||
eq(issueWatchdogs.status, "active"),
|
||||
))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
if (!watchdog) {
|
||||
return {
|
||||
kind: "invalid",
|
||||
detail: "Task-watchdog run context is not backed by an active persisted watchdog.",
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
kind: "watchdog",
|
||||
watchdogId: watchdog.id,
|
||||
companyId: watchdog.companyId,
|
||||
watchedIssueId: watchdog.issueId,
|
||||
watchdogIssueId: watchdog.watchdogIssueId ?? null,
|
||||
stopFingerprint: taskWatchdog.stopFingerprint,
|
||||
};
|
||||
}
|
||||
|
||||
export async function issueIsInTaskWatchdogSubtree(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
issueId: string,
|
||||
watchedIssueId: string,
|
||||
) {
|
||||
let currentId: string | null = issueId;
|
||||
const seen = new Set<string>();
|
||||
|
||||
for (let depth = 0; currentId && depth < MAX_WATCHDOG_SCOPE_ANCESTRY_DEPTH; depth += 1) {
|
||||
if (seen.has(currentId)) return false;
|
||||
seen.add(currentId);
|
||||
|
||||
const parent: { id: string; companyId: string; parentId: string | null; originKind: string | null } | null = await db
|
||||
.select({ id: issues.id, companyId: issues.companyId, parentId: issues.parentId, originKind: issues.originKind })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.id, currentId), eq(issues.companyId, companyId)))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!parent) return false;
|
||||
if (parent.originKind === TASK_WATCHDOG_ORIGIN_KIND) return false;
|
||||
if (currentId === watchedIssueId) return true;
|
||||
currentId = parent.parentId ?? null;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
export async function taskWatchdogScopeAllowsIssueMutation(
|
||||
db: Db,
|
||||
scope: TaskWatchdogMutationScope,
|
||||
issue: IssueScopeTarget,
|
||||
opts: { allowWatchdogIssue?: boolean } = {},
|
||||
) {
|
||||
if (scope.kind !== "watchdog") return scope;
|
||||
if (issue.companyId !== scope.companyId) {
|
||||
return {
|
||||
kind: "invalid" as const,
|
||||
detail: "Task-watchdog mutation target is outside the watchdog company.",
|
||||
};
|
||||
}
|
||||
if (opts.allowWatchdogIssue !== false && scope.watchdogIssueId && issue.id === scope.watchdogIssueId) {
|
||||
return scope;
|
||||
}
|
||||
if (await issueIsInTaskWatchdogSubtree(db, scope.companyId, issue.id, scope.watchedIssueId)) {
|
||||
return scope;
|
||||
}
|
||||
return {
|
||||
kind: "invalid" as const,
|
||||
detail: "Task-watchdog runs can only mutate the watched issue subtree.",
|
||||
};
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user