[codex] feat(watchdog): add task watchdog control plane (#8339)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The task lifecycle and recovery subsystems decide when agent work is still productive, stalled, or ready for review. > - Existing recovery paths can observe stopped or incomplete work, but there was no first-class per-task watchdog model with scoped review permissions. > - Watchdog follow-ups also need strict boundaries so recovery/status-only runs cannot mutate approvals or perform deliverable work. > - This pull request adds the task watchdog data model, API/service layer, scheduler/review flow, adapter wake context, UI configuration surfaces, and docs. > - The branch has been rebased onto current `paperclipai/paperclip` `master`; the watchdog migration is now ordered after master's latest migrations as `0104_issue_watchdogs`. > - The benefit is a more explicit task-review loop that preserves Paperclip's single-assignee and governance invariants while making stalled work easier to route. ## Linked Issues or Issue Description No linked GitHub issue. Paperclip task: [PAP-11275](/PAP/issues/PAP-11275). ## Problem or motivation Task recovery needs a first-class watchdog path that can inspect stopped work and create scoped follow-ups without bypassing normal task ownership. Board/UI users need a way to configure watchdogs on tasks and see watchdog-related live work. Recovery/status-only runs must remain limited to status reporting and must not create approvals, link approvals, or submit approval comments. ## Proposed solution Add a task-watchdog data model, scheduler/classifier, scoped mutation guard, adapter wake context, API/UI configuration surfaces, and documentation so watchdog agents can review stopped task subtrees under explicit boundaries. ## Alternatives considered Reuse the existing recovery-action flow only. That would keep stopped-work detection implicit, make per-task watchdog assignment harder to expose in the UI, and would not provide a durable scoped-review issue for stalled task trees. ## Roadmap alignment This is Paperclip control-plane lifecycle infrastructure for task execution and recovery. I checked `ROADMAP.md`; this PR does not duplicate an existing planned core item. ## What Changed - Added issue watchdog schema, migration, shared contracts, validators, CRUD API, and service support. - Added task watchdog scheduler/classifier behavior, scoped mutation enforcement, adapter wake context, and default watchdog mandate guidance. - Added UI surfaces for configuring watchdogs on new/existing tasks, viewing watchdog activity, and exposing the experimental setting. - Added docs for the user-facing task watchdog workflow and implementation semantics. - Gated new-task watchdog setup behind `enableTaskWatchdogs` and blocked cheap status-only recovery runs from approval mutations. - Rebased onto current `master` and renumbered the idempotent watchdog migration from the branch-local `0102_issue_watchdogs` slot to `0104_issue_watchdogs`. - Addressed Greptile feedback by loading watchdog classifier input with a recursive subtree query and centralizing the watchdog origin-kind constant. - Added and updated focused server/UI tests for watchdog routes, scheduler/classifier behavior, scope boundaries, live task visibility, settings, and new issue dialog behavior. ## Verification - `pnpm vitest run server/src/__tests__/task-watchdogs-scheduler.test.ts server/src/__tests__/task-watchdogs-classifier.test.ts` - `pnpm vitest run server/src/__tests__/approval-routes-idempotency.test.ts server/src/__tests__/issue-agent-mutation-ownership-routes.test.ts` - `pnpm vitest run ui/src/components/NewIssueDialog.test.tsx` - `pnpm --filter @paperclipai/server typecheck` - `git diff --check` - Verified the PR diff does not include `pnpm-lock.yaml` or `.github/workflows`. ## Risks - Medium risk: this introduces a new task lifecycle surface touching DB schema, server routes/services, adapter wake context, and UI task configuration. - Watchdog scheduling behavior depends on the new experimental setting and runtime context checks behaving consistently across local and production agents. - The watchdog migration is idempotent (`IF NOT EXISTS` / duplicate-object guards) so users who tried the previous branch-local migration number should not get duplicate-object failures. - CI and the second Greptile pass are pending after the latest review-fix push. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used OpenAI Codex, GPT-5-class coding agent in the Paperclip workspace. Exact runtime model id and context window were not exposed to the agent; tool use and local command execution were enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] If this change affects the UI, I have included before/after screenshots — N/A per Paperclip task instruction: do not add screenshots/images to this PR unless they are specifically part of the work. - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dotta
@@ -61,12 +61,32 @@ async function createApp(actorOverrides: Record<string, unknown> = {}) {
|
||||
};
|
||||
next();
|
||||
});
|
||||
app.use("/api", approvalRoutes({} as any));
|
||||
app.use("/api", approvalRoutes(createRouteDb()));
|
||||
app.use(errorHandler);
|
||||
return app;
|
||||
}
|
||||
|
||||
async function createAgentApp() {
|
||||
function createRouteDb(contextSnapshot: Record<string, unknown> = {}, runId = "run-1", agentId = "agent-1") {
|
||||
const runRows = [{
|
||||
id: runId,
|
||||
companyId: "company-1",
|
||||
agentId,
|
||||
contextSnapshot,
|
||||
}];
|
||||
return {
|
||||
select: vi.fn((selection: Record<string, unknown> = {}) => ({
|
||||
from: vi.fn(() => ({
|
||||
where: vi.fn(() => ({
|
||||
then: async (resolve: (rows: unknown[]) => unknown) => resolve(
|
||||
Object.keys(selection).includes("contextSnapshot") ? runRows : [],
|
||||
),
|
||||
})),
|
||||
})),
|
||||
})),
|
||||
} as any;
|
||||
}
|
||||
|
||||
async function createAgentApp(options: { runId?: string; contextSnapshot?: Record<string, unknown> } = {}) {
|
||||
const [{ errorHandler }, { approvalRoutes }] = await Promise.all([
|
||||
import("../middleware/index.js"),
|
||||
import("../routes/approvals.js"),
|
||||
@@ -78,12 +98,13 @@ async function createAgentApp() {
|
||||
type: "agent",
|
||||
agentId: "agent-1",
|
||||
companyId: "company-1",
|
||||
runId: options.runId ?? "run-1",
|
||||
source: "api_key",
|
||||
isInstanceAdmin: false,
|
||||
};
|
||||
next();
|
||||
});
|
||||
app.use("/api", approvalRoutes({} as any));
|
||||
app.use("/api", approvalRoutes(createRouteDb(options.contextSnapshot, options.runId ?? "run-1")));
|
||||
app.use(errorHandler);
|
||||
return app;
|
||||
}
|
||||
@@ -347,4 +368,80 @@ describe("approval routes idempotent retries", () => {
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("blocks status-only recovery runs from creating approvals", async () => {
|
||||
const res = await request(await createAgentApp({
|
||||
contextSnapshot: {
|
||||
modelProfile: "cheap",
|
||||
recoveryIntent: "status_only",
|
||||
allowDeliverableWork: false,
|
||||
allowDocumentUpdates: false,
|
||||
resumeRequiresNormalModel: true,
|
||||
},
|
||||
}))
|
||||
.post("/api/companies/company-1/approvals")
|
||||
.send({
|
||||
type: "request_board_approval",
|
||||
payload: { title: "Approve hosting spend" },
|
||||
});
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(403);
|
||||
expect(res.body.error).toContain("Cheap status-only recovery runs cannot create or modify approvals");
|
||||
expect(mockApprovalService.create).not.toHaveBeenCalled();
|
||||
expect(mockIssueApprovalService.linkManyForApproval).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("blocks status-only recovery runs from resubmitting approvals", async () => {
|
||||
mockApprovalService.getById.mockResolvedValue({
|
||||
id: "approval-7",
|
||||
companyId: "company-1",
|
||||
type: "request_board_approval",
|
||||
status: "revision_requested",
|
||||
payload: {},
|
||||
requestedByAgentId: "agent-1",
|
||||
});
|
||||
|
||||
const res = await request(await createAgentApp({
|
||||
contextSnapshot: {
|
||||
modelProfile: "cheap",
|
||||
recoveryIntent: "status_only",
|
||||
allowDeliverableWork: false,
|
||||
allowDocumentUpdates: false,
|
||||
resumeRequiresNormalModel: true,
|
||||
},
|
||||
}))
|
||||
.post("/api/approvals/approval-7/resubmit")
|
||||
.send({ payload: { title: "Retry" } });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(403);
|
||||
expect(res.body.error).toContain("Cheap status-only recovery runs cannot create or modify approvals");
|
||||
expect(mockApprovalService.resubmit).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("blocks status-only recovery runs from commenting on approvals", async () => {
|
||||
mockApprovalService.getById.mockResolvedValue({
|
||||
id: "approval-8",
|
||||
companyId: "company-1",
|
||||
type: "request_board_approval",
|
||||
status: "pending",
|
||||
payload: {},
|
||||
requestedByAgentId: "agent-1",
|
||||
});
|
||||
|
||||
const res = await request(await createAgentApp({
|
||||
contextSnapshot: {
|
||||
modelProfile: "cheap",
|
||||
recoveryIntent: "status_only",
|
||||
allowDeliverableWork: false,
|
||||
allowDocumentUpdates: false,
|
||||
resumeRequiresNormalModel: true,
|
||||
},
|
||||
}))
|
||||
.post("/api/approvals/approval-8/comments")
|
||||
.send({ body: "please approve" });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(403);
|
||||
expect(res.body.error).toContain("Cheap status-only recovery runs cannot create or modify approvals");
|
||||
expect(mockApprovalService.addComment).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -66,6 +66,7 @@ describe("instance settings routes", () => {
|
||||
enableIsolatedWorkspaces: false,
|
||||
enableIssuePlanDecompositions: false,
|
||||
enableExperimentalFileViewer: false,
|
||||
enableTaskWatchdogs: false,
|
||||
enableCloudSync: false,
|
||||
autoRestartDevServerWhenIdle: false,
|
||||
enableIssueGraphLivenessAutoRecovery: true,
|
||||
@@ -86,6 +87,7 @@ describe("instance settings routes", () => {
|
||||
enableIsolatedWorkspaces: true,
|
||||
enableIssuePlanDecompositions: true,
|
||||
enableExperimentalFileViewer: true,
|
||||
enableTaskWatchdogs: true,
|
||||
enableCloudSync: true,
|
||||
autoRestartDevServerWhenIdle: false,
|
||||
enableIssueGraphLivenessAutoRecovery: true,
|
||||
@@ -131,6 +133,7 @@ describe("instance settings routes", () => {
|
||||
enableIsolatedWorkspaces: false,
|
||||
enableIssuePlanDecompositions: false,
|
||||
enableExperimentalFileViewer: false,
|
||||
enableTaskWatchdogs: false,
|
||||
enableCloudSync: false,
|
||||
autoRestartDevServerWhenIdle: false,
|
||||
enableIssueGraphLivenessAutoRecovery: true,
|
||||
@@ -248,6 +251,43 @@ describe("instance settings routes", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("allows local board users to update task watchdog controls", async () => {
|
||||
const app = await createApp({
|
||||
type: "board",
|
||||
userId: "local-board",
|
||||
source: "local_implicit",
|
||||
isInstanceAdmin: true,
|
||||
});
|
||||
|
||||
await request(app)
|
||||
.patch("/api/instance/settings/experimental")
|
||||
.send({ enableTaskWatchdogs: true })
|
||||
.expect(200);
|
||||
|
||||
expect(mockInstanceSettingsService.updateExperimental).toHaveBeenCalledWith({
|
||||
enableTaskWatchdogs: true,
|
||||
});
|
||||
});
|
||||
|
||||
it("allows non-admin board users with company access to read but not update experimental settings", async () => {
|
||||
const app = await createApp({
|
||||
type: "board",
|
||||
userId: "user-1",
|
||||
source: "session",
|
||||
isInstanceAdmin: false,
|
||||
companyIds: ["company-1"],
|
||||
});
|
||||
|
||||
await request(app).get("/api/instance/settings/experimental").expect(200);
|
||||
|
||||
await request(app)
|
||||
.patch("/api/instance/settings/experimental")
|
||||
.send({ enableTaskWatchdogs: true })
|
||||
.expect(403);
|
||||
|
||||
expect(mockInstanceSettingsService.updateExperimental).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("allows local board users to read and update general settings", async () => {
|
||||
const app = await createApp({
|
||||
type: "board",
|
||||
|
||||
@@ -8,6 +8,7 @@ describe("instance settings service", () => {
|
||||
enableIsolatedWorkspaces: true,
|
||||
enableIssuePlanDecompositions: true,
|
||||
enableExperimentalFileViewer: true,
|
||||
enableTaskWatchdogs: true,
|
||||
enableCloudSync: true,
|
||||
autoRestartDevServerWhenIdle: true,
|
||||
enableIssueGraphLivenessAutoRecovery: true,
|
||||
@@ -20,6 +21,7 @@ describe("instance settings service", () => {
|
||||
enableConferenceRoomChat: false,
|
||||
enableIssuePlanDecompositions: true,
|
||||
enableExperimentalFileViewer: true,
|
||||
enableTaskWatchdogs: true,
|
||||
enableCloudSync: true,
|
||||
autoRestartDevServerWhenIdle: true,
|
||||
enableIssueGraphLivenessAutoRecovery: true,
|
||||
@@ -36,6 +38,14 @@ describe("instance settings service", () => {
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("defaults enableTaskWatchdogs to false for empty and legacy stored settings", () => {
|
||||
expect(normalizeExperimentalSettings(undefined).enableTaskWatchdogs).toBe(false);
|
||||
expect(normalizeExperimentalSettings({}).enableTaskWatchdogs).toBe(false);
|
||||
expect(
|
||||
normalizeExperimentalSettings({ enableExperimentalFileViewer: true }).enableTaskWatchdogs,
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("round-trips an enableConferenceRoomChat patch through the update merge", () => {
|
||||
// updateExperimental merges `{ ...normalize(current), ...patch }` and
|
||||
// re-normalizes; emulate that to prove the flag survives the roundtrip
|
||||
|
||||
@@ -15,6 +15,7 @@ const mockIssueService = vi.hoisted(() => ({
|
||||
assertCheckoutOwner: vi.fn(),
|
||||
create: vi.fn(),
|
||||
createChild: vi.fn(),
|
||||
decomposeAcceptedPlan: vi.fn(),
|
||||
getAttachmentById: vi.fn(),
|
||||
getByIdentifier: vi.fn(),
|
||||
getById: vi.fn(),
|
||||
@@ -66,12 +67,33 @@ const mockStorageService = vi.hoisted(() => ({
|
||||
const mockIssueThreadInteractionService = vi.hoisted(() => ({
|
||||
expireRequestConfirmationsSupersededByComment: vi.fn(async () => []),
|
||||
expireStaleRequestConfirmationsForIssueDocument: vi.fn(async () => []),
|
||||
listForIssue: vi.fn(async () => []),
|
||||
}));
|
||||
const mockIssueApprovalService = vi.hoisted(() => ({
|
||||
link: vi.fn(),
|
||||
unlink: vi.fn(),
|
||||
listApprovalsForIssue: vi.fn(async () => []),
|
||||
}));
|
||||
const mockIssueRecoveryActionService = vi.hoisted(() => ({
|
||||
getActiveForIssue: vi.fn(async () => null),
|
||||
listActiveForIssues: vi.fn(async () => new Map()),
|
||||
resolveActiveForIssue: vi.fn(async () => null),
|
||||
}));
|
||||
const mockTaskWatchdogService = vi.hoisted(() => ({
|
||||
getActiveForIssue: vi.fn(async () => null),
|
||||
revalidateMutationScope: vi.fn(async () => ({
|
||||
allowed: true,
|
||||
classification: { state: "stopped", stopFingerprint: "task_watchdog_stop:test" },
|
||||
})),
|
||||
reconcileForIssueAndAncestors: vi.fn(async () => ({
|
||||
checked: 0,
|
||||
triggered: 0,
|
||||
skipped: 0,
|
||||
watchdogIssueIds: [],
|
||||
})),
|
||||
upsertForIssue: vi.fn(),
|
||||
disableForIssue: vi.fn(async () => null),
|
||||
}));
|
||||
const mockHeartbeatService = vi.hoisted(() => ({
|
||||
wakeup: vi.fn(async () => undefined),
|
||||
reportRunActivity: vi.fn(async () => undefined),
|
||||
@@ -141,7 +163,7 @@ function registerRouteMocks() {
|
||||
})),
|
||||
listCompanyIds: vi.fn(async () => [companyId]),
|
||||
}),
|
||||
issueApprovalService: () => ({}),
|
||||
issueApprovalService: () => mockIssueApprovalService,
|
||||
issueRecoveryActionService: () => mockIssueRecoveryActionService,
|
||||
issueReferenceService: () => ({
|
||||
deleteDocumentSource: async () => undefined,
|
||||
@@ -158,6 +180,7 @@ function registerRouteMocks() {
|
||||
}),
|
||||
issueService: () => mockIssueService,
|
||||
issueThreadInteractionService: () => mockIssueThreadInteractionService,
|
||||
taskWatchdogService: () => mockTaskWatchdogService,
|
||||
logActivity: vi.fn(async () => undefined),
|
||||
projectService: () => ({}),
|
||||
routineService: () => ({
|
||||
@@ -344,6 +367,7 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
mockIssueService.assertCheckoutOwner.mockReset();
|
||||
mockIssueService.create.mockReset();
|
||||
mockIssueService.createChild.mockReset();
|
||||
mockIssueService.decomposeAcceptedPlan.mockReset();
|
||||
mockIssueService.getAttachmentById.mockReset();
|
||||
mockIssueService.getByIdentifier.mockReset();
|
||||
mockIssueService.getById.mockReset();
|
||||
@@ -385,6 +409,23 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
createdAt: new Date("2026-05-13T17:55:00.000Z"),
|
||||
updatedAt: new Date("2026-05-13T18:05:00.000Z"),
|
||||
});
|
||||
mockTaskWatchdogService.getActiveForIssue.mockReset();
|
||||
mockTaskWatchdogService.getActiveForIssue.mockResolvedValue(null);
|
||||
mockTaskWatchdogService.revalidateMutationScope.mockReset();
|
||||
mockTaskWatchdogService.revalidateMutationScope.mockResolvedValue({
|
||||
allowed: true,
|
||||
classification: { state: "stopped", stopFingerprint: "task_watchdog_stop:test" },
|
||||
});
|
||||
mockTaskWatchdogService.reconcileForIssueAndAncestors.mockReset();
|
||||
mockTaskWatchdogService.reconcileForIssueAndAncestors.mockResolvedValue({
|
||||
checked: 0,
|
||||
triggered: 0,
|
||||
skipped: 0,
|
||||
watchdogIssueIds: [],
|
||||
});
|
||||
mockTaskWatchdogService.upsertForIssue.mockReset();
|
||||
mockTaskWatchdogService.disableForIssue.mockReset();
|
||||
mockTaskWatchdogService.disableForIssue.mockResolvedValue(null);
|
||||
mockHeartbeatService.wakeup.mockReset();
|
||||
mockHeartbeatService.wakeup.mockResolvedValue(undefined);
|
||||
mockHeartbeatService.reportRunActivity.mockReset();
|
||||
@@ -395,6 +436,12 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
mockHeartbeatService.getActiveRunForAgent.mockResolvedValue(null);
|
||||
mockHeartbeatService.cancelRun.mockReset();
|
||||
mockHeartbeatService.cancelRun.mockResolvedValue(null);
|
||||
mockIssueApprovalService.link.mockReset();
|
||||
mockIssueApprovalService.unlink.mockReset();
|
||||
mockIssueApprovalService.listApprovalsForIssue.mockReset();
|
||||
mockIssueApprovalService.listApprovalsForIssue.mockResolvedValue([]);
|
||||
mockIssueThreadInteractionService.listForIssue.mockReset();
|
||||
mockIssueThreadInteractionService.listForIssue.mockResolvedValue([]);
|
||||
mockIssueService.remove.mockReset();
|
||||
mockIssueService.removeAttachment.mockReset();
|
||||
mockIssueService.update.mockReset();
|
||||
@@ -447,6 +494,27 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
},
|
||||
parentBlockerAdded: false,
|
||||
}));
|
||||
mockIssueService.decomposeAcceptedPlan.mockImplementation(async (_sourceIssueId: string, input: Record<string, unknown>) => {
|
||||
const children = input.children as Record<string, unknown>[];
|
||||
return {
|
||||
decomposition: {
|
||||
id: "decomposition-1",
|
||||
status: "completed",
|
||||
childIssueIds: children.map((child) => child.id),
|
||||
},
|
||||
childIssueIds: children.map((child) => child.id),
|
||||
newlyCreatedIssues: children.map((child) => ({
|
||||
...makeIssue({
|
||||
id: child.id,
|
||||
parentId: issueId,
|
||||
status: child.status,
|
||||
assigneeAgentId: child.assigneeAgentId ?? null,
|
||||
}),
|
||||
...child,
|
||||
companyId,
|
||||
})),
|
||||
};
|
||||
});
|
||||
mockIssueService.getRelationSummaries.mockResolvedValue({ blockedBy: [], blocks: [] });
|
||||
mockIssueService.listWakeableBlockedDependents.mockResolvedValue([]);
|
||||
mockIssueService.getWakeableParentAfterChildCompletion.mockResolvedValue(null);
|
||||
@@ -727,9 +795,18 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
provider: "test",
|
||||
title: "Artifact",
|
||||
}),
|
||||
"Cheap status-only recovery runs cannot update issue documents",
|
||||
],
|
||||
[
|
||||
"work product update",
|
||||
(app: express.Express) => request(app).patch("/api/work-products/product-1").send({ title: "Blocked" }),
|
||||
"Cheap status-only recovery runs cannot update issue documents",
|
||||
],
|
||||
[
|
||||
"work product delete",
|
||||
(app: express.Express) => request(app).delete("/api/work-products/product-1"),
|
||||
"Cheap status-only recovery runs cannot update issue documents",
|
||||
],
|
||||
["work product update", (app: express.Express) => request(app).patch("/api/work-products/product-1").send({ title: "Blocked" })],
|
||||
["work product delete", (app: express.Express) => request(app).delete("/api/work-products/product-1")],
|
||||
[
|
||||
"low-trust promotion",
|
||||
(app: express.Express) =>
|
||||
@@ -739,6 +816,7 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
title: "Promoted artifact",
|
||||
summary: "Sanitized output",
|
||||
}),
|
||||
"Cheap status-only recovery runs cannot update issue documents",
|
||||
],
|
||||
[
|
||||
"attachment upload",
|
||||
@@ -746,9 +824,28 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
request(app)
|
||||
.post(`/api/companies/${companyId}/issues/${issueId}/attachments`)
|
||||
.attach("file", Buffer.from("report"), { filename: "report.txt", contentType: "text/plain" }),
|
||||
"Cheap status-only recovery runs cannot update issue documents",
|
||||
],
|
||||
["attachment delete", (app: express.Express) => request(app).delete("/api/attachments/attachment-1")],
|
||||
])("blocks cheap status-only recovery runs from %s", async (_name, sendRequest) => {
|
||||
[
|
||||
"attachment delete",
|
||||
(app: express.Express) => request(app).delete("/api/attachments/attachment-1"),
|
||||
"Cheap status-only recovery runs cannot update issue documents",
|
||||
],
|
||||
[
|
||||
"issue approval link",
|
||||
(app: express.Express) =>
|
||||
request(app).post(`/api/issues/${issueId}/approvals`).send({
|
||||
approvalId: "88888888-8888-4888-8888-888888888888",
|
||||
}),
|
||||
"Cheap status-only recovery runs cannot create or modify approvals",
|
||||
],
|
||||
[
|
||||
"issue approval unlink",
|
||||
(app: express.Express) =>
|
||||
request(app).delete(`/api/issues/${issueId}/approvals/88888888-8888-4888-8888-888888888888`),
|
||||
"Cheap status-only recovery runs cannot create or modify approvals",
|
||||
],
|
||||
])("blocks cheap status-only recovery runs from %s", async (_name, sendRequest, expectedError) => {
|
||||
const app = await createApp(
|
||||
ownerActor(),
|
||||
createRunContextDb({
|
||||
@@ -763,7 +860,7 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
const res = await sendRequest(app);
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(403);
|
||||
expect(res.body.error).toContain("Cheap status-only recovery runs cannot update issue documents");
|
||||
expect(res.body.error).toContain(expectedError);
|
||||
expect(mockIssueService.assertCheckoutOwner).toHaveBeenCalledWith(issueId, ownerAgentId, ownerRunId);
|
||||
expect(mockWorkProductService.createForIssue).not.toHaveBeenCalled();
|
||||
expect(mockWorkProductService.update).not.toHaveBeenCalled();
|
||||
@@ -771,6 +868,8 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
expect(mockStorageService.putFile).not.toHaveBeenCalled();
|
||||
expect(mockStorageService.deleteObject).not.toHaveBeenCalled();
|
||||
expect(mockIssueService.removeAttachment).not.toHaveBeenCalled();
|
||||
expect(mockIssueApprovalService.link).not.toHaveBeenCalled();
|
||||
expect(mockIssueApprovalService.unlink).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it.each([
|
||||
@@ -1147,4 +1246,363 @@ describe("agent issue mutation checkout ownership", () => {
|
||||
}));
|
||||
expect(mockIssueService.update).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
describe("task watchdog scope grants", () => {
|
||||
const watchdogRunId = "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaab";
|
||||
const watchdogReportIssueId = "cccccccc-cccc-4ccc-8ccc-cccccccccccd";
|
||||
|
||||
// The watchdog agent (peerAgentId) is NOT the assignee of the watched issue
|
||||
// (ownerAgentId), so the base authorization boundary (issue:mutate) denies.
|
||||
// The watchdog scope must grant the mutation regardless.
|
||||
function watchdogActor(runId: string = watchdogRunId) {
|
||||
return {
|
||||
type: "agent",
|
||||
agentId: peerAgentId,
|
||||
companyId,
|
||||
source: "agent_key",
|
||||
runId,
|
||||
};
|
||||
}
|
||||
|
||||
function createWatchdogDb(options: {
|
||||
watchedIssueId?: string;
|
||||
watchdogIssueId?: string | null;
|
||||
ancestryParentId?: string | null;
|
||||
watchdogRows?: Record<string, unknown>[];
|
||||
} = {}) {
|
||||
const watchedIssueId = options.watchedIssueId ?? issueId;
|
||||
const runRows = [{
|
||||
id: watchdogRunId,
|
||||
companyId,
|
||||
agentId: peerAgentId,
|
||||
contextSnapshot: { taskWatchdog: { watchedIssueId, stopFingerprint: "task_watchdog_stop:test" } },
|
||||
}];
|
||||
const watchdogRows = options.watchdogRows ?? [{
|
||||
id: "dddddddd-dddd-4ddd-8ddd-ddddddddddde",
|
||||
companyId,
|
||||
issueId: watchedIssueId,
|
||||
watchdogAgentId: peerAgentId,
|
||||
watchdogIssueId: options.watchdogIssueId ?? watchdogReportIssueId,
|
||||
status: "active",
|
||||
}];
|
||||
const ancestryRows = [{
|
||||
id: "ancestry",
|
||||
companyId,
|
||||
parentId: options.ancestryParentId ?? null,
|
||||
}];
|
||||
const rowsForSelection = (selection: Record<string, unknown>) => {
|
||||
const keys = Object.keys(selection);
|
||||
if (keys.includes("entityId")) return [];
|
||||
if (keys.includes("contextSnapshot")) return runRows;
|
||||
if (keys.includes("watchdogAgentId")) return watchdogRows;
|
||||
if (keys.includes("parentId")) return ancestryRows;
|
||||
if (keys.includes("status")) return [];
|
||||
if (keys.includes("agentCompanyId")) return runRows;
|
||||
return [{ id: peerAgentId, companyId, permissions: {}, role: "engineer", reportsTo: null }];
|
||||
};
|
||||
const buildQuery = (selection: Record<string, unknown>) => {
|
||||
const whereResult = {
|
||||
orderBy: vi.fn(async () => []),
|
||||
then: async (resolve: (rows: unknown[]) => unknown) => resolve(rowsForSelection(selection)),
|
||||
};
|
||||
const query = {
|
||||
innerJoin: vi.fn(() => query),
|
||||
where: vi.fn(() => whereResult),
|
||||
};
|
||||
return query;
|
||||
};
|
||||
return {
|
||||
transaction: async (callback: (tx: Record<string, never>) => Promise<unknown>) => callback({}),
|
||||
select: vi.fn((selection: Record<string, unknown> = {}) => ({
|
||||
from: vi.fn(() => buildQuery(selection)),
|
||||
})),
|
||||
};
|
||||
}
|
||||
|
||||
// The base boundary always denies a cross-agent issue:mutate; only the
|
||||
// watchdog scope can widen access. Denying it here proves the grant works.
|
||||
function denyBaseBoundary() {
|
||||
mockAccessService.decide.mockImplementation(async (input: { action: string }) => ({
|
||||
allowed: input.action === "company_scope:read" || input.action === "issue:read" || input.action === "tasks:assign",
|
||||
action: input.action,
|
||||
reason:
|
||||
input.action === "company_scope:read" || input.action === "issue:read" || input.action === "tasks:assign"
|
||||
? "allow_explicit_grant"
|
||||
: "deny_missing_grant",
|
||||
explanation: "Watchdog test boundary default.",
|
||||
}));
|
||||
}
|
||||
|
||||
it("lets a watchdog run comment on a watched issue assigned to a different agent", async () => {
|
||||
denyBaseBoundary();
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue({ assigneeAgentId: ownerAgentId }));
|
||||
|
||||
const app = await createApp(watchdogActor(), createWatchdogDb());
|
||||
const res = await request(app).post(`/api/issues/${issueId}/comments`).send({ body: "Watchdog finding" });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(201);
|
||||
expect(mockIssueService.addComment).toHaveBeenCalledWith(
|
||||
issueId,
|
||||
"Watchdog finding",
|
||||
expect.any(Object),
|
||||
expect.any(Object),
|
||||
);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["in_progress"],
|
||||
["blocked"],
|
||||
["todo"],
|
||||
])("lets a watchdog run transition a watched issue to %s", async (status) => {
|
||||
denyBaseBoundary();
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue({ status: "in_progress", assigneeAgentId: ownerAgentId }));
|
||||
mockIssueService.update.mockImplementation(async (_id: string, patch: Record<string, unknown>) => ({
|
||||
...makeIssue({ assigneeAgentId: ownerAgentId }),
|
||||
...patch,
|
||||
}));
|
||||
|
||||
const app = await createApp(watchdogActor(), createWatchdogDb());
|
||||
const res = await request(app).patch(`/api/issues/${issueId}`).send({ status });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockIssueService.update).toHaveBeenCalledWith(issueId, expect.objectContaining({ status }));
|
||||
});
|
||||
|
||||
it("lets a watchdog run transition a watched issue to in_review with a live review path", async () => {
|
||||
denyBaseBoundary();
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue({ status: "in_progress", assigneeAgentId: ownerAgentId }));
|
||||
mockIssueService.update.mockImplementation(async (_id: string, patch: Record<string, unknown>) => ({
|
||||
...makeIssue({ assigneeAgentId: ownerAgentId }),
|
||||
...patch,
|
||||
}));
|
||||
// A pending interaction is a valid review path, so the agent in_review guard
|
||||
// is satisfied — this isolates the test to the watchdog boundary grant.
|
||||
mockIssueThreadInteractionService.listForIssue.mockResolvedValue([{ status: "pending" }] as never);
|
||||
|
||||
const app = await createApp(watchdogActor(), createWatchdogDb());
|
||||
const res = await request(app).patch(`/api/issues/${issueId}`).send({ status: "in_review" });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockIssueService.update).toHaveBeenCalledWith(issueId, expect.objectContaining({ status: "in_review" }));
|
||||
});
|
||||
|
||||
it("rejects stale watchdog source mutations when revalidation finds a live path", async () => {
|
||||
denyBaseBoundary();
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue({ status: "in_progress", assigneeAgentId: ownerAgentId }));
|
||||
mockTaskWatchdogService.revalidateMutationScope.mockResolvedValueOnce({
|
||||
allowed: false,
|
||||
reason:
|
||||
"Task-watchdog review is stale because the watched subtree now has a live, waiting, already-reviewed, or not-applicable path; refresh the source state before mutating it.",
|
||||
classification: { state: "live", liveIssueIds: [issueId] },
|
||||
});
|
||||
|
||||
const app = await createApp(watchdogActor(), createWatchdogDb());
|
||||
const res = await request(app).patch(`/api/issues/${issueId}`).send({ status: "blocked" });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(409);
|
||||
expect(res.body.error).toContain("Task-watchdog review is stale");
|
||||
expect(mockIssueService.update).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("suppresses watchdog follow-up creation when current source revalidation is live", async () => {
|
||||
denyBaseBoundary();
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue({ assigneeAgentId: ownerAgentId }));
|
||||
mockTaskWatchdogService.revalidateMutationScope.mockResolvedValueOnce({
|
||||
allowed: false,
|
||||
reason:
|
||||
"Task-watchdog review is stale because the watched subtree now has a live, waiting, already-reviewed, or not-applicable path; refresh the source state before mutating it.",
|
||||
classification: { state: "live", liveIssueIds: [issueId] },
|
||||
});
|
||||
|
||||
const app = await createApp(watchdogActor(), createWatchdogDb());
|
||||
const res = await request(app)
|
||||
.post(`/api/issues/${issueId}/children`)
|
||||
.send({ title: "Stale follow-up", status: "todo" });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(409);
|
||||
expect(res.body.error).toContain("Task-watchdog review is stale");
|
||||
expect(mockIssueService.createChild).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("serializes watchdog accepted-plan follow-ups behind one active child lane", async () => {
|
||||
denyBaseBoundary();
|
||||
mockIssueService.list.mockResolvedValue([]);
|
||||
mockAgentService.resolveByReference.mockImplementation(async (_companyId: string, reference: string) => ({
|
||||
ambiguous: false,
|
||||
agent: reference === ownerAgentId ? makeAgent(ownerAgentId) : null,
|
||||
}));
|
||||
mockIssueService.getById.mockImplementation(async (id: string) => {
|
||||
if (id === watchdogReportIssueId) {
|
||||
return makeIssue({
|
||||
id: watchdogReportIssueId,
|
||||
originKind: "task_watchdog",
|
||||
status: "in_progress",
|
||||
assigneeAgentId: peerAgentId,
|
||||
});
|
||||
}
|
||||
return makeIssue({ assigneeAgentId: ownerAgentId });
|
||||
});
|
||||
|
||||
const app = await createApp(watchdogActor(), createWatchdogDb());
|
||||
const res = await request(app)
|
||||
.post(`/api/issues/${issueId}/accepted-plan-decompositions`)
|
||||
.send({
|
||||
acceptedPlanRevisionId: "aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa",
|
||||
children: [
|
||||
{ title: "Fix watchdog authorization", assigneeAgentId: ownerAgentId },
|
||||
{ title: "Fix watchdog startup race", assigneeAgentId: ownerAgentId },
|
||||
],
|
||||
});
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
const decompositionInput = mockIssueService.decomposeAcceptedPlan.mock.calls[0]?.[1];
|
||||
const children = decompositionInput.children as Array<Record<string, unknown>>;
|
||||
expect(children).toHaveLength(2);
|
||||
expect(children[0]).toEqual(expect.objectContaining({
|
||||
title: "Fix watchdog authorization",
|
||||
status: "todo",
|
||||
assigneeAgentId: ownerAgentId,
|
||||
}));
|
||||
expect(children[1]).toEqual(expect.objectContaining({
|
||||
title: "Fix watchdog startup race",
|
||||
status: "blocked",
|
||||
assigneeAgentId: ownerAgentId,
|
||||
blockedByIssueIds: [children[0]?.id],
|
||||
}));
|
||||
expect(mockHeartbeatService.wakeup).toHaveBeenCalledTimes(1);
|
||||
expect(mockHeartbeatService.wakeup).toHaveBeenCalledWith(
|
||||
ownerAgentId,
|
||||
expect.objectContaining({
|
||||
payload: expect.objectContaining({ issueId: children[0]?.id }),
|
||||
}),
|
||||
);
|
||||
expect(mockIssueService.update).toHaveBeenCalledWith(
|
||||
watchdogReportIssueId,
|
||||
expect.objectContaining({
|
||||
status: "blocked",
|
||||
blockedByIssueIds: [children[0]?.id],
|
||||
actorAgentId: peerAgentId,
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("preserves normal accepted-plan decomposition parallel wakeups outside watchdog context", async () => {
|
||||
mockAgentService.resolveByReference.mockImplementation(async (_companyId: string, reference: string) => ({
|
||||
ambiguous: false,
|
||||
agent: reference === ownerAgentId ? makeAgent(ownerAgentId) : null,
|
||||
}));
|
||||
const app = await createApp(ownerActor());
|
||||
const res = await request(app)
|
||||
.post(`/api/issues/${issueId}/accepted-plan-decompositions`)
|
||||
.send({
|
||||
acceptedPlanRevisionId: "bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb",
|
||||
children: [
|
||||
{ title: "Implement backend", assigneeAgentId: ownerAgentId },
|
||||
{ title: "Implement frontend", assigneeAgentId: ownerAgentId },
|
||||
],
|
||||
});
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
const decompositionInput = mockIssueService.decomposeAcceptedPlan.mock.calls[0]?.[1];
|
||||
const children = decompositionInput.children as Array<Record<string, unknown>>;
|
||||
expect(children).toHaveLength(2);
|
||||
expect(children[0]).toEqual(expect.objectContaining({ status: "todo" }));
|
||||
expect(children[1]).toEqual(expect.objectContaining({ status: "todo" }));
|
||||
expect(children[1]?.blockedByIssueIds).toBeUndefined();
|
||||
expect(mockHeartbeatService.wakeup).toHaveBeenCalledTimes(2);
|
||||
expect(mockHeartbeatService.wakeup).toHaveBeenNthCalledWith(
|
||||
1,
|
||||
ownerAgentId,
|
||||
expect.objectContaining({
|
||||
payload: expect.objectContaining({ issueId: children[0]?.id }),
|
||||
}),
|
||||
);
|
||||
expect(mockHeartbeatService.wakeup).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
ownerAgentId,
|
||||
expect.objectContaining({
|
||||
payload: expect.objectContaining({ issueId: children[1]?.id }),
|
||||
}),
|
||||
);
|
||||
expect(mockIssueService.update).not.toHaveBeenCalledWith(
|
||||
watchdogReportIssueId,
|
||||
expect.anything(),
|
||||
);
|
||||
});
|
||||
|
||||
it("lets a watchdog run reassign a watched issue to an active same-company agent", async () => {
|
||||
denyBaseBoundary();
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue({ assigneeAgentId: ownerAgentId }));
|
||||
mockIssueService.update.mockImplementation(async (_id: string, patch: Record<string, unknown>) => ({
|
||||
...makeIssue({ assigneeAgentId: ownerAgentId }),
|
||||
...patch,
|
||||
}));
|
||||
mockAgentService.resolveByReference.mockResolvedValue({ ambiguous: false, agent: makeAgent(peerAgentId) });
|
||||
|
||||
const app = await createApp(watchdogActor(), createWatchdogDb());
|
||||
const res = await request(app).patch(`/api/issues/${issueId}`).send({ assigneeAgentId: peerAgentId });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(200);
|
||||
expect(mockIssueService.update).toHaveBeenCalledWith(
|
||||
issueId,
|
||||
expect.objectContaining({ assigneeAgentId: peerAgentId }),
|
||||
);
|
||||
});
|
||||
|
||||
it("still denies a watchdog run mutating an issue outside the watched subtree", async () => {
|
||||
denyBaseBoundary();
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue({ assigneeAgentId: ownerAgentId }));
|
||||
|
||||
// The watched issue is a different issue, and the target's ancestry chain
|
||||
// (parentId === null) never reaches it, so it is outside the subtree.
|
||||
const outsideWatched = "eeeeeeee-eeee-4eee-8eee-eeeeeeeeeeef";
|
||||
const app = await createApp(
|
||||
watchdogActor(),
|
||||
createWatchdogDb({ watchedIssueId: outsideWatched, ancestryParentId: null }),
|
||||
);
|
||||
const res = await request(app).patch(`/api/issues/${issueId}`).send({ status: "blocked" });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(403);
|
||||
expect(res.body.error).toBe("Task-watchdog runs can only mutate the watched issue subtree.");
|
||||
expect(mockIssueService.update).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("still enforces normal assignment guards for watchdog reassignment", async () => {
|
||||
// Base boundary denied AND tasks:assign denied: the watchdog grant lets the
|
||||
// mutation past the ownership boundary, but the assignment guard must still bite.
|
||||
mockAccessService.decide.mockImplementation(async (input: { action: string }) => ({
|
||||
allowed: input.action === "company_scope:read",
|
||||
action: input.action,
|
||||
reason: input.action === "company_scope:read" ? "allow_explicit_grant" : "deny_policy_restricted",
|
||||
explanation:
|
||||
input.action === "tasks:assign"
|
||||
? "Target agent requires approval before task assignment."
|
||||
: "Watchdog test boundary default.",
|
||||
}));
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue({ assigneeAgentId: ownerAgentId }));
|
||||
mockAgentService.resolveByReference.mockResolvedValue({ ambiguous: false, agent: makeAgent(peerAgentId) });
|
||||
|
||||
const app = await createApp(watchdogActor(), createWatchdogDb());
|
||||
const res = await request(app).patch(`/api/issues/${issueId}`).send({ assigneeAgentId: peerAgentId });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(403);
|
||||
expect(res.body.error).toContain("requires approval");
|
||||
expect(mockIssueService.update).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("denies an invalid watchdog run context even when the base boundary would allow it", async () => {
|
||||
// Run context claims a watched issue, but no active persisted watchdog backs it.
|
||||
const app = await createApp(
|
||||
watchdogActor(),
|
||||
createWatchdogDb({ watchdogRows: [] }),
|
||||
);
|
||||
mockIssueService.getById.mockResolvedValue(makeIssue({ assigneeAgentId: peerAgentId }));
|
||||
|
||||
const res = await request(app).patch(`/api/issues/${issueId}`).send({ status: "blocked" });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(403);
|
||||
expect(res.body.error).toBe("Task-watchdog run context is not backed by an active persisted watchdog.");
|
||||
expect(mockIssueService.update).not.toHaveBeenCalled();
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -112,6 +112,11 @@ function registerModuleMocks() {
|
||||
}),
|
||||
issueService: () => mockIssueService,
|
||||
issueThreadInteractionService: () => mockInteractionService,
|
||||
taskWatchdogService: () => ({
|
||||
getActiveForIssue: vi.fn(async () => null),
|
||||
upsertForIssue: vi.fn(),
|
||||
disableForIssue: vi.fn(async () => null),
|
||||
}),
|
||||
logActivity: mockLogActivity,
|
||||
projectService: () => ({}),
|
||||
routineService: () => ({
|
||||
|
||||
@@ -0,0 +1,632 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import express from "express";
|
||||
import request from "supertest";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
activityLog,
|
||||
agentWakeupRequests,
|
||||
agents,
|
||||
companies,
|
||||
companyMemberships,
|
||||
createDb,
|
||||
heartbeatRunEvents,
|
||||
heartbeatRuns,
|
||||
issueComments,
|
||||
issueRelations,
|
||||
issueWatchdogs,
|
||||
issues,
|
||||
principalPermissionGrants,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { errorHandler } from "../middleware/index.js";
|
||||
import { issueRoutes } from "../routes/issues.js";
|
||||
import { ensureHumanRoleDefaultGrants } from "../services/principal-access-compatibility.js";
|
||||
import { taskWatchdogService } from "../services/task-watchdogs.js";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping embedded Postgres issue watchdog route tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
describeEmbeddedPostgres("issue watchdog routes", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-issue-watchdogs-routes-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await db.delete(activityLog);
|
||||
await db.delete(issueComments);
|
||||
await db.delete(heartbeatRunEvents);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(issueRelations);
|
||||
await db.delete(issueWatchdogs);
|
||||
await db.delete(issues);
|
||||
await db.delete(agents);
|
||||
await db.delete(principalPermissionGrants);
|
||||
await db.delete(companyMemberships);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
function createApp(companyId: string, actor?: Record<string, unknown>) {
|
||||
const app = express();
|
||||
app.use(express.json());
|
||||
app.use((req, _res, next) => {
|
||||
(req as any).actor = actor ?? {
|
||||
type: "board",
|
||||
userId: "cloud-user-1",
|
||||
companyIds: [companyId],
|
||||
memberships: [{ companyId, membershipRole: "owner", status: "active" }],
|
||||
source: "cloud_tenant",
|
||||
isInstanceAdmin: false,
|
||||
};
|
||||
next();
|
||||
});
|
||||
app.use("/api", issueRoutes(db, {} as any, { taskWatchdogEnqueueWakeup: null }));
|
||||
app.use(errorHandler);
|
||||
return app;
|
||||
}
|
||||
|
||||
function uniqueIssuePrefix() {
|
||||
return `W${randomUUID().replace(/-/g, "").slice(0, 5).toUpperCase()}`;
|
||||
}
|
||||
|
||||
async function seedCloudTenantMember(companyId: string) {
|
||||
await db.insert(companyMemberships).values({
|
||||
companyId,
|
||||
principalType: "user",
|
||||
principalId: "cloud-user-1",
|
||||
status: "active",
|
||||
membershipRole: "owner",
|
||||
updatedAt: new Date(),
|
||||
});
|
||||
await ensureHumanRoleDefaultGrants(db, {
|
||||
companyId,
|
||||
principalId: "cloud-user-1",
|
||||
membershipRole: "owner",
|
||||
grantedByUserId: null,
|
||||
});
|
||||
}
|
||||
|
||||
async function seedCompany(name = "Paperclip") {
|
||||
const companyId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name,
|
||||
issuePrefix: uniqueIssuePrefix(),
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
await seedCloudTenantMember(companyId);
|
||||
return companyId;
|
||||
}
|
||||
|
||||
async function seedAgent(companyId: string, overrides: Partial<typeof agents.$inferInsert> = {}) {
|
||||
const id = overrides.id ?? randomUUID();
|
||||
await db.insert(agents).values({
|
||||
id,
|
||||
companyId,
|
||||
name: overrides.name ?? "Watchdog Agent",
|
||||
role: overrides.role ?? "engineer",
|
||||
status: overrides.status ?? "active",
|
||||
adapterType: overrides.adapterType ?? "codex_local",
|
||||
adapterConfig: overrides.adapterConfig ?? {},
|
||||
runtimeConfig: overrides.runtimeConfig ?? {},
|
||||
permissions: overrides.permissions ?? {},
|
||||
reportsTo: overrides.reportsTo,
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
async function seedIssue(companyId: string, overrides: Partial<typeof issues.$inferInsert> = {}) {
|
||||
const id = overrides.id ?? randomUUID();
|
||||
await db.insert(issues).values({
|
||||
id,
|
||||
companyId,
|
||||
title: overrides.title ?? "Watched task",
|
||||
status: overrides.status ?? "todo",
|
||||
priority: overrides.priority ?? "medium",
|
||||
identifier: overrides.identifier,
|
||||
issueNumber: overrides.issueNumber,
|
||||
assigneeAgentId: overrides.assigneeAgentId,
|
||||
parentId: overrides.parentId,
|
||||
projectId: overrides.projectId,
|
||||
goalId: overrides.goalId,
|
||||
originKind: overrides.originKind,
|
||||
originId: overrides.originId,
|
||||
// Default to an "established" issue (created before the first-run grace
|
||||
// window) so attaching a watchdog evaluates immediately instead of being
|
||||
// deferred by the pending-first-run guard.
|
||||
createdAt: overrides.createdAt ?? new Date(Date.now() - 60 * 60 * 1000),
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
async function seedWatchdogRun(input: {
|
||||
companyId: string;
|
||||
watchdogAgentId: string;
|
||||
watchedIssueId: string;
|
||||
watchdogIssueId: string;
|
||||
}) {
|
||||
await db.insert(issueWatchdogs).values({
|
||||
companyId: input.companyId,
|
||||
issueId: input.watchedIssueId,
|
||||
watchdogAgentId: input.watchdogAgentId,
|
||||
watchdogIssueId: input.watchdogIssueId,
|
||||
status: "active",
|
||||
});
|
||||
await taskWatchdogService(db).reconcileTaskWatchdogs({ companyId: input.companyId });
|
||||
const [watchdog] = await db
|
||||
.select({ lastObservedFingerprint: issueWatchdogs.lastObservedFingerprint })
|
||||
.from(issueWatchdogs)
|
||||
.where(and(eq(issueWatchdogs.companyId, input.companyId), eq(issueWatchdogs.issueId, input.watchedIssueId)));
|
||||
const runId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: runId,
|
||||
companyId: input.companyId,
|
||||
agentId: input.watchdogAgentId,
|
||||
status: "running",
|
||||
contextSnapshot: {
|
||||
issueId: input.watchdogIssueId,
|
||||
taskWatchdog: {
|
||||
watchedIssueId: input.watchedIssueId,
|
||||
watchedIssueIdentifier: "WDOG-ROOT",
|
||||
watchedIssueTitle: "Watched root",
|
||||
stopFingerprint: watchdog?.lastObservedFingerprint,
|
||||
},
|
||||
},
|
||||
});
|
||||
return runId;
|
||||
}
|
||||
|
||||
async function waitForAssignmentWakeup(companyId: string) {
|
||||
for (let attempt = 0; attempt < 20; attempt += 1) {
|
||||
const rows = await db
|
||||
.select({ id: heartbeatRuns.id })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.companyId, companyId))
|
||||
.limit(1);
|
||||
if (rows.length > 0) return;
|
||||
await new Promise((resolve) => setTimeout(resolve, 10));
|
||||
}
|
||||
}
|
||||
|
||||
it("creates, updates, reads, lists, and removes an issue watchdog with activity logs", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const issueId = await seedIssue(companyId, { identifier: "WDOG-1", issueNumber: 1 });
|
||||
const firstAgentId = await seedAgent(companyId, { name: "First Watchdog" });
|
||||
const secondAgentId = await seedAgent(companyId, { name: "Second Watchdog" });
|
||||
const app = createApp(companyId);
|
||||
|
||||
const created = await request(app)
|
||||
.put(`/api/issues/${issueId}/watchdog`)
|
||||
.send({ agentId: firstAgentId, instructions: "Check screenshots and tests." });
|
||||
|
||||
expect(created.status, JSON.stringify(created.body)).toBe(200);
|
||||
expect(created.body).toMatchObject({
|
||||
issueId,
|
||||
watchdogAgentId: firstAgentId,
|
||||
instructions: "Check screenshots and tests.",
|
||||
status: "active",
|
||||
});
|
||||
|
||||
const updated = await request(app)
|
||||
.put(`/api/issues/${issueId}/watchdog`)
|
||||
.send({ agentId: secondAgentId, instructions: "Be skeptical." });
|
||||
|
||||
expect(updated.status, JSON.stringify(updated.body)).toBe(200);
|
||||
expect(updated.body.id).toBe(created.body.id);
|
||||
expect(updated.body).toMatchObject({
|
||||
issueId,
|
||||
watchdogAgentId: secondAgentId,
|
||||
instructions: "Be skeptical.",
|
||||
status: "active",
|
||||
});
|
||||
|
||||
const read = await request(app).get(`/api/issues/${issueId}/watchdog`);
|
||||
expect(read.status, JSON.stringify(read.body)).toBe(200);
|
||||
expect(read.body).toMatchObject({ id: created.body.id, watchdogAgentId: secondAgentId });
|
||||
|
||||
const detail = await request(app).get(`/api/issues/${issueId}`);
|
||||
expect(detail.status, JSON.stringify(detail.body)).toBe(200);
|
||||
expect(detail.body.watchdog).toMatchObject({ id: created.body.id, watchdogAgentId: secondAgentId });
|
||||
|
||||
const list = await request(app).get(`/api/companies/${companyId}/issues`);
|
||||
expect(list.status, JSON.stringify(list.body)).toBe(200);
|
||||
expect(list.body.find((issue: { id: string }) => issue.id === issueId)?.watchdog)
|
||||
.toMatchObject({ id: created.body.id, watchdogAgentId: secondAgentId });
|
||||
|
||||
const removed = await request(app).delete(`/api/issues/${issueId}/watchdog`);
|
||||
expect(removed.status, JSON.stringify(removed.body)).toBe(200);
|
||||
expect(removed.body).toEqual({ ok: true });
|
||||
|
||||
const afterDelete = await request(app).get(`/api/issues/${issueId}/watchdog`);
|
||||
expect(afterDelete.status, JSON.stringify(afterDelete.body)).toBe(200);
|
||||
expect(afterDelete.body).toBeNull();
|
||||
|
||||
const stored = await db
|
||||
.select()
|
||||
.from(issueWatchdogs)
|
||||
.where(and(eq(issueWatchdogs.companyId, companyId), eq(issueWatchdogs.issueId, issueId)))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
expect(stored).toMatchObject({
|
||||
id: created.body.id,
|
||||
status: "disabled",
|
||||
watchdogAgentId: secondAgentId,
|
||||
});
|
||||
|
||||
const actions = await db
|
||||
.select({ action: activityLog.action })
|
||||
.from(activityLog)
|
||||
.where(eq(activityLog.entityId, issueId));
|
||||
const actionNames = actions.map((row) => row.action);
|
||||
expect(actionNames.filter((action) => action.startsWith("issue.watchdog_"))).toEqual([
|
||||
"issue.watchdog_created",
|
||||
"issue.watchdog_updated",
|
||||
"issue.watchdog_removed",
|
||||
]);
|
||||
expect(actionNames).toContain("issue.task_watchdog_triggered");
|
||||
});
|
||||
|
||||
it("handles concurrent first-time watchdog upserts without duplicate-key failures", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const issueId = await seedIssue(companyId, { identifier: "WDOG-RACE", issueNumber: 99 });
|
||||
const agentId = await seedAgent(companyId, { name: "Race Watchdog" });
|
||||
const app = createApp(companyId);
|
||||
|
||||
const responses = await Promise.all(
|
||||
Array.from({ length: 12 }, (_, index) =>
|
||||
request(app)
|
||||
.put(`/api/issues/${issueId}/watchdog`)
|
||||
.send({ agentId, instructions: `Concurrent instructions ${index}` }),
|
||||
),
|
||||
);
|
||||
|
||||
expect(responses.map((res) => res.status), JSON.stringify(responses.map((res) => res.body)))
|
||||
.toEqual(Array(12).fill(200));
|
||||
const stored = await db
|
||||
.select()
|
||||
.from(issueWatchdogs)
|
||||
.where(and(eq(issueWatchdogs.companyId, companyId), eq(issueWatchdogs.issueId, issueId)));
|
||||
expect(stored).toHaveLength(1);
|
||||
expect(stored[0]).toMatchObject({ status: "active", watchdogAgentId: agentId });
|
||||
});
|
||||
|
||||
it("creates an issue and watchdog atomically from the create issue route", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent(companyId);
|
||||
const app = createApp(companyId);
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/companies/${companyId}/issues`)
|
||||
.send({
|
||||
title: "Create with watchdog",
|
||||
watchdog: {
|
||||
agentId,
|
||||
instructions: "Confirm the final state.",
|
||||
},
|
||||
});
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(201);
|
||||
expect(res.body.watchdog).toMatchObject({
|
||||
issueId: res.body.id,
|
||||
watchdogAgentId: agentId,
|
||||
instructions: "Confirm the final state.",
|
||||
status: "active",
|
||||
});
|
||||
|
||||
const rows = await db
|
||||
.select()
|
||||
.from(issueWatchdogs)
|
||||
.where(eq(issueWatchdogs.issueId, res.body.id));
|
||||
expect(rows).toHaveLength(1);
|
||||
|
||||
const activityRows = await db
|
||||
.select({ action: activityLog.action })
|
||||
.from(activityLog)
|
||||
.where(eq(activityLog.entityId, res.body.id));
|
||||
expect(activityRows.map((row) => row.action)).toContain("issue.watchdog_created");
|
||||
});
|
||||
|
||||
it("does not create an immediate watchdog review for a newly assigned issue", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const workerAgentId = await seedAgent(companyId, { name: "Worker" });
|
||||
const watchdogAgentId = await seedAgent(companyId, { name: "Watchdog" });
|
||||
const app = createApp(companyId);
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/companies/${companyId}/issues`)
|
||||
.send({
|
||||
title: "Assigned issue with watchdog",
|
||||
assigneeAgentId: workerAgentId,
|
||||
watchdog: {
|
||||
agentId: watchdogAgentId,
|
||||
instructions: "Confirm whether the worker got started.",
|
||||
},
|
||||
});
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(201);
|
||||
await waitForAssignmentWakeup(companyId);
|
||||
expect(res.body).toMatchObject({
|
||||
assigneeAgentId: workerAgentId,
|
||||
watchdog: {
|
||||
issueId: res.body.id,
|
||||
watchdogAgentId,
|
||||
status: "active",
|
||||
},
|
||||
});
|
||||
|
||||
const watchdogReviewIssues = await db
|
||||
.select()
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "task_watchdog")));
|
||||
expect(watchdogReviewIssues).toHaveLength(0);
|
||||
|
||||
const [watchdog] = await db
|
||||
.select()
|
||||
.from(issueWatchdogs)
|
||||
.where(and(eq(issueWatchdogs.companyId, companyId), eq(issueWatchdogs.issueId, res.body.id)));
|
||||
expect(watchdog?.triggerCount).toBe(0);
|
||||
|
||||
const taskWatchdogActivity = await db
|
||||
.select({ action: activityLog.action })
|
||||
.from(activityLog)
|
||||
.where(and(eq(activityLog.entityId, res.body.id), eq(activityLog.action, "issue.task_watchdog_triggered")));
|
||||
expect(taskWatchdogActivity).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("enforces persisted watchdog scope for issue mutations and child creation", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const watchdogAgentId = await seedAgent(companyId, { name: "Scoped Watchdog" });
|
||||
const watchedRootId = await seedIssue(companyId, { title: "Watched root", identifier: "WDOG-ROOT" });
|
||||
const watchedChildId = await seedIssue(companyId, { title: "Watched child", parentId: watchedRootId });
|
||||
const unrelatedRootId = await seedIssue(companyId, { title: "Unrelated root" });
|
||||
const watchdogIssueId = await seedIssue(companyId, {
|
||||
title: "Reusable watchdog issue",
|
||||
parentId: watchedRootId,
|
||||
assigneeAgentId: watchdogAgentId,
|
||||
originKind: "task_watchdog",
|
||||
originId: watchedRootId,
|
||||
});
|
||||
const watchdogIssueChildId = await seedIssue(companyId, {
|
||||
title: "Watchdog issue child",
|
||||
parentId: watchdogIssueId,
|
||||
});
|
||||
const runId = await seedWatchdogRun({
|
||||
companyId,
|
||||
watchdogAgentId,
|
||||
watchedIssueId: watchedRootId,
|
||||
watchdogIssueId,
|
||||
});
|
||||
const app = createApp(companyId, {
|
||||
type: "agent",
|
||||
agentId: watchdogAgentId,
|
||||
companyId,
|
||||
runId,
|
||||
source: "agent_jwt",
|
||||
});
|
||||
|
||||
const watchdogIssuePatch = await request(app)
|
||||
.patch(`/api/issues/${watchdogIssueId}`)
|
||||
.send({ title: "Reusable watchdog issue completed" });
|
||||
expect(watchdogIssuePatch.status, JSON.stringify(watchdogIssuePatch.body)).toBe(200);
|
||||
|
||||
const deniedWatchdogDescendantPatch = await request(app)
|
||||
.patch(`/api/issues/${watchdogIssueChildId}`)
|
||||
.send({ title: "Denied watchdog descendant mutation" });
|
||||
expect(deniedWatchdogDescendantPatch.status, JSON.stringify(deniedWatchdogDescendantPatch.body)).toBe(403);
|
||||
expect(deniedWatchdogDescendantPatch.body.error).toBe(
|
||||
"Task-watchdog runs can only mutate the watched issue subtree.",
|
||||
);
|
||||
|
||||
const deniedPatch = await request(app)
|
||||
.patch(`/api/issues/${unrelatedRootId}`)
|
||||
.send({ title: "Out-of-scope mutation" });
|
||||
expect(deniedPatch.status, JSON.stringify(deniedPatch.body)).toBe(403);
|
||||
expect(deniedPatch.body.error).toBe("Task-watchdog runs can only mutate the watched issue subtree.");
|
||||
|
||||
const deniedChild = await request(app)
|
||||
.post(`/api/issues/${unrelatedRootId}/children`)
|
||||
.send({ title: "Denied unrelated child" });
|
||||
expect(deniedChild.status, JSON.stringify(deniedChild.body)).toBe(403);
|
||||
expect(deniedChild.body.error).toBe("Task-watchdog runs can only mutate the watched issue subtree.");
|
||||
|
||||
const deniedWatchdogIssueChild = await request(app)
|
||||
.post(`/api/issues/${watchdogIssueId}/children`)
|
||||
.send({ title: "Denied watchdog issue child" });
|
||||
expect(deniedWatchdogIssueChild.status, JSON.stringify(deniedWatchdogIssueChild.body)).toBe(403);
|
||||
const deniedVisibleProbeIssues = await db
|
||||
.select({ id: issues.id })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.title, "Denied watchdog issue child")));
|
||||
expect(deniedVisibleProbeIssues).toHaveLength(0);
|
||||
|
||||
const deniedParentCreate = await request(app)
|
||||
.post(`/api/companies/${companyId}/issues`)
|
||||
.send({ title: "Denied parent create", parentId: unrelatedRootId });
|
||||
expect(deniedParentCreate.status, JSON.stringify(deniedParentCreate.body)).toBe(403);
|
||||
expect(deniedParentCreate.body.error).toBe("Task-watchdog runs can only mutate the watched issue subtree.");
|
||||
|
||||
const deniedNestedWatchdog = await request(app)
|
||||
.put(`/api/issues/${watchedChildId}/watchdog`)
|
||||
.send({ agentId: watchdogAgentId, instructions: "Create a nested watchdog" });
|
||||
expect(deniedNestedWatchdog.status, JSON.stringify(deniedNestedWatchdog.body)).toBe(403);
|
||||
expect(deniedNestedWatchdog.body.error).toBe("Task-watchdog runs cannot change watchdog configuration.");
|
||||
|
||||
const deniedWatchdogRemoval = await request(app).delete(`/api/issues/${watchedRootId}/watchdog`);
|
||||
expect(deniedWatchdogRemoval.status, JSON.stringify(deniedWatchdogRemoval.body)).toBe(403);
|
||||
expect(deniedWatchdogRemoval.body.error).toBe("Task-watchdog runs cannot change watchdog configuration.");
|
||||
|
||||
const nestedWatchdogs = await db
|
||||
.select({ id: issueWatchdogs.id })
|
||||
.from(issueWatchdogs)
|
||||
.where(and(eq(issueWatchdogs.companyId, companyId), eq(issueWatchdogs.issueId, watchedChildId)));
|
||||
expect(nestedWatchdogs).toHaveLength(0);
|
||||
|
||||
const allowedChild = await request(app)
|
||||
.post(`/api/issues/${watchedChildId}/children`)
|
||||
.send({ title: "Allowed watched child" });
|
||||
expect(allowedChild.status, JSON.stringify(allowedChild.body)).toBe(201);
|
||||
expect(allowedChild.body.parentId).toBe(watchedChildId);
|
||||
});
|
||||
|
||||
it("routes watchdog-discovered product bugs outside the watched source tree with evidence links", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const watchdogAgentId = await seedAgent(companyId, { name: "Product Bug Watchdog" });
|
||||
const watchedRootId = await seedIssue(companyId, {
|
||||
title: "Watched root",
|
||||
identifier: "PAP-100",
|
||||
issueNumber: 100,
|
||||
});
|
||||
const watchedChildId = await seedIssue(companyId, {
|
||||
title: "Watched child",
|
||||
identifier: "PAP-101",
|
||||
issueNumber: 101,
|
||||
parentId: watchedRootId,
|
||||
});
|
||||
const watchdogIssueId = await seedIssue(companyId, {
|
||||
title: "Reusable watchdog issue",
|
||||
identifier: "PAP-102",
|
||||
issueNumber: 102,
|
||||
parentId: watchedRootId,
|
||||
assigneeAgentId: watchdogAgentId,
|
||||
originKind: "task_watchdog",
|
||||
originId: watchedRootId,
|
||||
});
|
||||
const runId = await seedWatchdogRun({
|
||||
companyId,
|
||||
watchdogAgentId,
|
||||
watchedIssueId: watchedRootId,
|
||||
watchdogIssueId,
|
||||
});
|
||||
const app = createApp(companyId, {
|
||||
type: "agent",
|
||||
agentId: watchdogAgentId,
|
||||
companyId,
|
||||
runId,
|
||||
source: "agent_jwt",
|
||||
});
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/companies/${companyId}/issues`)
|
||||
.send({
|
||||
title: "Fix watchdog source-tree pollution",
|
||||
description: "Watchdog found a Paperclip follow-up routing bug.",
|
||||
parentId: watchedChildId,
|
||||
watchdogDiscovery: {
|
||||
kind: "product_bug",
|
||||
evidenceMarkdown: "The watchdog would otherwise create this under the watched child.",
|
||||
},
|
||||
});
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(201);
|
||||
expect(res.body).toMatchObject({
|
||||
title: "Fix watchdog source-tree pollution",
|
||||
parentId: null,
|
||||
originKind: "task_watchdog_product_bug",
|
||||
originId: watchedRootId,
|
||||
originRunId: runId,
|
||||
});
|
||||
expect(res.body.description).toContain("## Watchdog Discovery");
|
||||
expect(res.body.description).toContain("Watched source issue: [PAP-100](/PAP/issues/PAP-100)");
|
||||
expect(res.body.description).toContain("Watchdog issue: [PAP-102](/PAP/issues/PAP-102)");
|
||||
expect(res.body.referencedIssueIdentifiers).toEqual(expect.arrayContaining(["PAP-100", "PAP-102"]));
|
||||
|
||||
const watchedSourceChildren = await db
|
||||
.select({ id: issues.id, title: issues.title })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.parentId, watchedChildId)));
|
||||
expect(watchedSourceChildren).toHaveLength(0);
|
||||
|
||||
const [createdActivity] = await db
|
||||
.select({ details: activityLog.details })
|
||||
.from(activityLog)
|
||||
.where(and(eq(activityLog.companyId, companyId), eq(activityLog.entityId, res.body.id)));
|
||||
expect(createdActivity?.details).toMatchObject({
|
||||
watchdogDiscovery: {
|
||||
kind: "product_bug",
|
||||
sourceIssueId: watchedRootId,
|
||||
watchdogIssueId,
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
it("rejects watchdog interaction-resolution attempts outside the persisted watched subtree", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const watchdogAgentId = await seedAgent(companyId, { name: "Interaction Watchdog" });
|
||||
const watchedRootId = await seedIssue(companyId, { title: "Watched root" });
|
||||
const unrelatedRootId = await seedIssue(companyId, { title: "Unrelated root" });
|
||||
const watchdogIssueId = await seedIssue(companyId, { title: "Reusable watchdog issue" });
|
||||
const runId = await seedWatchdogRun({
|
||||
companyId,
|
||||
watchdogAgentId,
|
||||
watchedIssueId: watchedRootId,
|
||||
watchdogIssueId,
|
||||
});
|
||||
const app = createApp(companyId, {
|
||||
type: "agent",
|
||||
agentId: watchdogAgentId,
|
||||
companyId,
|
||||
runId,
|
||||
source: "agent_jwt",
|
||||
});
|
||||
|
||||
const res = await request(app)
|
||||
.post(`/api/issues/${unrelatedRootId}/interactions/${randomUUID()}/accept`)
|
||||
.send({});
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(403);
|
||||
expect(res.body.error).toBe("Task-watchdog runs can only mutate the watched issue subtree.");
|
||||
});
|
||||
|
||||
it("rejects cross-company watched issues and watchdog agents", async () => {
|
||||
const companyId = await seedCompany("Allowed company");
|
||||
const otherCompanyId = await seedCompany("Other company");
|
||||
const issueId = await seedIssue(companyId);
|
||||
const otherIssueId = await seedIssue(otherCompanyId);
|
||||
const otherAgentId = await seedAgent(otherCompanyId);
|
||||
const app = createApp(companyId);
|
||||
|
||||
const foreignIssue = await request(app)
|
||||
.put(`/api/issues/${otherIssueId}/watchdog`)
|
||||
.send({ agentId: otherAgentId });
|
||||
expect(foreignIssue.status, JSON.stringify(foreignIssue.body)).toBe(403);
|
||||
|
||||
const foreignAgent = await request(app)
|
||||
.put(`/api/issues/${issueId}/watchdog`)
|
||||
.send({ agentId: otherAgentId });
|
||||
expect(foreignAgent.status, JSON.stringify(foreignAgent.body)).toBe(404);
|
||||
});
|
||||
|
||||
it.each(["paused", "terminated", "pending_approval"])(
|
||||
"rejects %s watchdog agents",
|
||||
async (status) => {
|
||||
const companyId = await seedCompany();
|
||||
const issueId = await seedIssue(companyId);
|
||||
const agentId = await seedAgent(companyId, { status });
|
||||
const app = createApp(companyId);
|
||||
|
||||
const res = await request(app)
|
||||
.put(`/api/issues/${issueId}/watchdog`)
|
||||
.send({ agentId });
|
||||
|
||||
expect(res.status, JSON.stringify(res.body)).toBe(409);
|
||||
expect(res.body.error).toBe("Cannot assign watchdog to an agent that is not invokable");
|
||||
},
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,205 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { classifyTaskWatchdogSubtree, type TaskWatchdogClassifierIssue } from "../services/task-watchdogs.ts";
|
||||
|
||||
const companyId = "company-1";
|
||||
const sourceId = "source-1";
|
||||
const childId = "child-1";
|
||||
const watchdogId = "watchdog-1";
|
||||
|
||||
function issue(overrides: Partial<TaskWatchdogClassifierIssue> = {}): TaskWatchdogClassifierIssue {
|
||||
return {
|
||||
id: sourceId,
|
||||
companyId,
|
||||
identifier: "PAP-1",
|
||||
title: "Source",
|
||||
status: "todo",
|
||||
parentId: null,
|
||||
assigneeAgentId: "agent-1",
|
||||
assigneeUserId: null,
|
||||
originKind: "manual",
|
||||
updatedAt: new Date("2026-06-17T20:00:00.000Z"),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function classify(overrides: Partial<Parameters<typeof classifyTaskWatchdogSubtree>[0]> = {}) {
|
||||
return classifyTaskWatchdogSubtree({
|
||||
watchdog: {
|
||||
companyId,
|
||||
issueId: sourceId,
|
||||
lastReviewedFingerprint: null,
|
||||
},
|
||||
issues: [issue()],
|
||||
...overrides,
|
||||
});
|
||||
}
|
||||
|
||||
describe("task watchdog subtree classifier", () => {
|
||||
it("suppresses watchdog wakeups while watched subtree work has a live path", () => {
|
||||
const result = classify({
|
||||
issues: [
|
||||
issue(),
|
||||
issue({ id: childId, identifier: "PAP-2", parentId: sourceId, status: "in_progress" }),
|
||||
],
|
||||
activeRuns: [{ companyId, issueId: childId, agentId: "agent-1", status: "running" }],
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
state: "live",
|
||||
liveIssueIds: [childId],
|
||||
});
|
||||
});
|
||||
|
||||
it("treats terminal and waiting leaves as stopped work that needs verification", () => {
|
||||
const result = classify({
|
||||
issues: [
|
||||
issue({ status: "done" }),
|
||||
issue({ id: childId, identifier: "PAP-2", parentId: sourceId, status: "in_review" }),
|
||||
],
|
||||
pendingInteractions: [{ companyId, issueId: childId, id: "interaction-1", status: "pending" }],
|
||||
});
|
||||
|
||||
expect(result.state).toBe("stopped");
|
||||
if (result.state !== "stopped") return;
|
||||
expect(result.stopFingerprint).toMatch(/^task_watchdog_stop:/);
|
||||
expect(result.stoppedLeaves).toEqual([
|
||||
expect.objectContaining({
|
||||
issueId: childId,
|
||||
status: "in_review",
|
||||
pendingInteractionIds: ["interaction-1"],
|
||||
}),
|
||||
]);
|
||||
});
|
||||
|
||||
it("suppresses an unchanged stopped fingerprint once the watchdog reviewed it", () => {
|
||||
const stopped = classify({
|
||||
issues: [issue({ status: "blocked" })],
|
||||
});
|
||||
expect(stopped.state).toBe("stopped");
|
||||
if (stopped.state !== "stopped") return;
|
||||
|
||||
const reviewed = classify({
|
||||
watchdog: {
|
||||
companyId,
|
||||
issueId: sourceId,
|
||||
lastReviewedFingerprint: stopped.stopFingerprint,
|
||||
},
|
||||
issues: [issue({ status: "blocked" })],
|
||||
});
|
||||
|
||||
expect(reviewed).toMatchObject({
|
||||
state: "already_reviewed",
|
||||
stopFingerprint: stopped.stopFingerprint,
|
||||
});
|
||||
});
|
||||
|
||||
it("excludes task-watchdog issues and their descendants from watched subtree scans", () => {
|
||||
const result = classify({
|
||||
issues: [
|
||||
issue({ status: "done" }),
|
||||
issue({
|
||||
id: watchdogId,
|
||||
identifier: "PAP-3",
|
||||
title: "Watchdog",
|
||||
parentId: sourceId,
|
||||
originKind: "task_watchdog",
|
||||
status: "in_progress",
|
||||
}),
|
||||
issue({
|
||||
id: "watchdog-child-1",
|
||||
identifier: "PAP-4",
|
||||
title: "Nested watchdog work",
|
||||
parentId: watchdogId,
|
||||
originKind: "manual",
|
||||
status: "in_progress",
|
||||
}),
|
||||
],
|
||||
activeRuns: [{ companyId, issueId: "watchdog-child-1", agentId: "agent-1", status: "running" }],
|
||||
});
|
||||
|
||||
expect(result.state).toBe("stopped");
|
||||
expect(result.includedIssueIds).toEqual([sourceId]);
|
||||
});
|
||||
|
||||
it("defers a stopped verdict for an issue created inside the first-run grace window", () => {
|
||||
const createdAt = new Date("2026-06-18T16:32:45.731Z");
|
||||
const result = classify({
|
||||
issues: [issue({ status: "todo", createdAt })],
|
||||
// Evaluation races the issue's own assignment run ~100ms after creation.
|
||||
evaluatedAt: new Date("2026-06-18T16:32:45.835Z"),
|
||||
firstRunGraceMs: 15_000,
|
||||
});
|
||||
|
||||
expect(result.state).toBe("pending_first_run");
|
||||
if (result.state !== "pending_first_run") return;
|
||||
expect(result.pendingIssueIds).toEqual([sourceId]);
|
||||
});
|
||||
|
||||
it("does not defer when a recently-created issue already completed a run", () => {
|
||||
const createdAt = new Date("2026-06-18T16:32:45.731Z");
|
||||
const result = classify({
|
||||
issues: [issue({ status: "blocked", createdAt })],
|
||||
evaluatedAt: new Date("2026-06-18T16:32:48.000Z"),
|
||||
firstRunGraceMs: 15_000,
|
||||
completedRunIssueIds: [sourceId],
|
||||
});
|
||||
|
||||
expect(result.state).toBe("stopped");
|
||||
});
|
||||
|
||||
it("treats a queued assignment run inside the create-race window as live", () => {
|
||||
const createdAt = new Date("2026-06-18T16:32:45.731Z");
|
||||
const result = classify({
|
||||
issues: [issue({ status: "todo", createdAt })],
|
||||
activeRuns: [{ companyId, issueId: sourceId, agentId: "agent-1", status: "queued" }],
|
||||
evaluatedAt: new Date("2026-06-18T16:32:45.835Z"),
|
||||
firstRunGraceMs: 15_000,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({ state: "live", liveIssueIds: [sourceId] });
|
||||
});
|
||||
|
||||
it("treats a queued assignment wake inside the create-race window as live", () => {
|
||||
const createdAt = new Date("2026-06-18T16:32:45.731Z");
|
||||
const result = classify({
|
||||
issues: [issue({ status: "todo", createdAt })],
|
||||
queuedWakeRequests: [{ companyId, issueId: sourceId, agentId: "agent-1", status: "queued" }],
|
||||
evaluatedAt: new Date("2026-06-18T16:32:45.835Z"),
|
||||
firstRunGraceMs: 15_000,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({ state: "live", liveIssueIds: [sourceId] });
|
||||
});
|
||||
|
||||
it("triggers a genuinely idle assigned issue once the grace window has elapsed", () => {
|
||||
const createdAt = new Date("2026-06-18T16:32:45.731Z");
|
||||
const result = classify({
|
||||
issues: [issue({ status: "todo", createdAt })],
|
||||
// 60s later: no run, no wake, past the grace window.
|
||||
evaluatedAt: new Date("2026-06-18T16:33:45.731Z"),
|
||||
firstRunGraceMs: 15_000,
|
||||
});
|
||||
|
||||
expect(result.state).toBe("stopped");
|
||||
});
|
||||
|
||||
it("does not evaluate a task-watchdog issue as a watched source", () => {
|
||||
const result = classify({
|
||||
watchdog: {
|
||||
companyId,
|
||||
issueId: watchdogId,
|
||||
lastReviewedFingerprint: null,
|
||||
},
|
||||
issues: [
|
||||
issue({
|
||||
id: watchdogId,
|
||||
identifier: "PAP-3",
|
||||
title: "Watchdog",
|
||||
originKind: "task_watchdog",
|
||||
}),
|
||||
],
|
||||
});
|
||||
|
||||
expect(result.state).toBe("not_applicable");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,592 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
activityLog,
|
||||
agentWakeupRequests,
|
||||
agents,
|
||||
companies,
|
||||
createDb,
|
||||
documents,
|
||||
heartbeatRuns,
|
||||
issueComments,
|
||||
issueDocuments,
|
||||
issueWorkProducts,
|
||||
issues,
|
||||
issueWatchdogs,
|
||||
} from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import { taskWatchdogService } from "../services/task-watchdogs.ts";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping embedded Postgres task watchdog scheduler tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
describeEmbeddedPostgres("task watchdog scheduler", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-task-watchdogs-scheduler-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
await db.delete(activityLog);
|
||||
await db.delete(issueWorkProducts);
|
||||
await db.delete(issueDocuments);
|
||||
await db.delete(documents);
|
||||
await db.delete(issueComments);
|
||||
await db.delete(heartbeatRuns);
|
||||
await db.delete(agentWakeupRequests);
|
||||
await db.delete(issueWatchdogs);
|
||||
await db.delete(issues);
|
||||
await db.delete(agents);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
async function seedCompany() {
|
||||
const companyId = randomUUID();
|
||||
await db.insert(companies).values({
|
||||
id: companyId,
|
||||
name: "Watchdog Co",
|
||||
issuePrefix: `WD${randomUUID().replace(/-/g, "").slice(0, 4).toUpperCase()}`,
|
||||
issueCounter: 0,
|
||||
requireBoardApprovalForNewAgents: false,
|
||||
});
|
||||
return companyId;
|
||||
}
|
||||
|
||||
async function seedAgent(companyId: string, overrides: Partial<typeof agents.$inferInsert> = {}) {
|
||||
const id = overrides.id ?? randomUUID();
|
||||
await db.insert(agents).values({
|
||||
id,
|
||||
companyId,
|
||||
name: overrides.name ?? "Watchdog Agent",
|
||||
role: overrides.role ?? "engineer",
|
||||
status: overrides.status ?? "active",
|
||||
adapterType: overrides.adapterType ?? "codex_local",
|
||||
adapterConfig: overrides.adapterConfig ?? {},
|
||||
runtimeConfig: overrides.runtimeConfig ?? {},
|
||||
permissions: overrides.permissions ?? {},
|
||||
reportsTo: overrides.reportsTo,
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
async function seedIssue(companyId: string, overrides: Partial<typeof issues.$inferInsert> = {}) {
|
||||
const id = overrides.id ?? randomUUID();
|
||||
await db.insert(issues).values({
|
||||
id,
|
||||
companyId,
|
||||
title: overrides.title ?? "Watched issue",
|
||||
status: overrides.status ?? "done",
|
||||
priority: overrides.priority ?? "medium",
|
||||
identifier: overrides.identifier ?? `WDOG-${Math.floor(Math.random() * 10_000)}`,
|
||||
issueNumber: overrides.issueNumber ?? Math.floor(Math.random() * 10_000),
|
||||
parentId: overrides.parentId,
|
||||
assigneeAgentId: overrides.assigneeAgentId,
|
||||
originKind: overrides.originKind,
|
||||
originId: overrides.originId,
|
||||
originFingerprint: overrides.originFingerprint,
|
||||
updatedAt: overrides.updatedAt,
|
||||
// Default to an "established" issue (created well before the first-run
|
||||
// grace window) so the pending-first-run guard does not defer it. Tests
|
||||
// exercising the create-race pass an explicit recent `createdAt`.
|
||||
createdAt: overrides.createdAt ?? new Date(Date.now() - 60 * 60 * 1000),
|
||||
});
|
||||
return id;
|
||||
}
|
||||
|
||||
async function seedIssueDocument(companyId: string, issueId: string, updatedAt: Date) {
|
||||
const [document] = await db.insert(documents).values({
|
||||
companyId,
|
||||
title: "Plan",
|
||||
latestBody: "Plan body",
|
||||
updatedAt,
|
||||
}).returning();
|
||||
await db.insert(issueDocuments).values({
|
||||
companyId,
|
||||
issueId,
|
||||
documentId: document!.id,
|
||||
key: "plan",
|
||||
updatedAt,
|
||||
});
|
||||
}
|
||||
|
||||
async function seedIssueWorkProduct(companyId: string, issueId: string, updatedAt: Date) {
|
||||
await db.insert(issueWorkProducts).values({
|
||||
companyId,
|
||||
issueId,
|
||||
type: "artifact",
|
||||
provider: "test",
|
||||
title: "Report",
|
||||
status: "ready",
|
||||
updatedAt,
|
||||
});
|
||||
}
|
||||
|
||||
async function seedWatchdog(companyId: string, issueId: string, agentId: string) {
|
||||
const [row] = await db.insert(issueWatchdogs).values({
|
||||
companyId,
|
||||
issueId,
|
||||
watchdogAgentId: agentId,
|
||||
instructions: "Verify stopped work.",
|
||||
status: "active",
|
||||
}).returning();
|
||||
return row;
|
||||
}
|
||||
|
||||
function createService() {
|
||||
const wakes: Array<{ agentId: string; opts: Record<string, unknown> | undefined }> = [];
|
||||
const service = taskWatchdogService(db, {
|
||||
enqueueWakeup: async (agentId, opts) => {
|
||||
wakes.push({ agentId, opts });
|
||||
return { id: randomUUID() };
|
||||
},
|
||||
});
|
||||
return { service, wakes };
|
||||
}
|
||||
|
||||
it("creates one reusable watchdog issue and wakes the watchdog on the initial stopped state", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-1", status: "done" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 1 });
|
||||
expect(wakes).toHaveLength(1);
|
||||
expect(wakes[0]?.agentId).toBe(agentId);
|
||||
expect(wakes[0]?.opts?.reason).toBe("task_watchdog_stopped_subtree");
|
||||
expect(wakes[0]?.opts?.contextSnapshot).toMatchObject({
|
||||
taskWatchdog: {
|
||||
watchedIssueId: sourceId,
|
||||
watchedIssueIdentifier: "WDOG-1",
|
||||
capabilities: {
|
||||
targetScope: {
|
||||
watchedIssueId: sourceId,
|
||||
includeNonWatchdogDescendants: true,
|
||||
excludedOriginKinds: ["task_watchdog"],
|
||||
},
|
||||
operations: expect.arrayContaining([
|
||||
"comment_on_watched_subtree_issues",
|
||||
"create_child_issues_under_non_watchdog_watched_subtree",
|
||||
"create_product_bug_followups_outside_watched_subtree",
|
||||
"update_reusable_watchdog_issue",
|
||||
]),
|
||||
deniedOperations: expect.arrayContaining([
|
||||
"create_visible_probe_issues_or_throwaway_tasks",
|
||||
"create_product_bug_followups_as_source_tree_children",
|
||||
"mutate_task_watchdog_descendants",
|
||||
]),
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
const watchdogIssues = await db
|
||||
.select()
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "task_watchdog")));
|
||||
expect(watchdogIssues).toHaveLength(1);
|
||||
expect(watchdogIssues[0]).toMatchObject({
|
||||
parentId: sourceId,
|
||||
originId: sourceId,
|
||||
assigneeAgentId: agentId,
|
||||
status: "todo",
|
||||
});
|
||||
|
||||
const [watchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
expect(watchdog?.watchdogIssueId).toBe(watchdogIssues[0]?.id);
|
||||
expect(watchdog?.lastObservedFingerprint).toMatch(/^task_watchdog_stop:/);
|
||||
expect(watchdog?.triggerCount).toBe(1);
|
||||
});
|
||||
|
||||
it("does not trigger while a non-watchdog descendant has live work", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-2", status: "in_progress" });
|
||||
const childId = await seedIssue(companyId, { parentId: sourceId, status: "in_progress" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
await db.insert(heartbeatRuns).values({
|
||||
companyId,
|
||||
agentId,
|
||||
status: "queued",
|
||||
invocationSource: "assignment",
|
||||
contextSnapshot: { issueId: childId },
|
||||
});
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 0, live: 1 });
|
||||
expect(wakes).toHaveLength(0);
|
||||
const watchdogIssues = await db
|
||||
.select({ id: issues.id })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "task_watchdog")));
|
||||
expect(watchdogIssues).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("does not trigger while a descendant has a queued assignment wake", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-WAKE", status: "in_progress" });
|
||||
const childId = await seedIssue(companyId, { parentId: sourceId, status: "todo" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
await db.insert(agentWakeupRequests).values({
|
||||
companyId,
|
||||
agentId,
|
||||
status: "queued",
|
||||
source: "assignment",
|
||||
triggerDetail: "system",
|
||||
reason: "issue_assigned",
|
||||
payload: { issueId: childId },
|
||||
});
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 0, live: 1 });
|
||||
expect(wakes).toHaveLength(0);
|
||||
const watchdogIssues = await db
|
||||
.select({ id: issues.id })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "task_watchdog")));
|
||||
expect(watchdogIssues).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("does not keep the source live for runs under a nested task-watchdog issue", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-NEST", status: "done" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
const nestedWatchdogIssueId = await seedIssue(companyId, {
|
||||
parentId: sourceId,
|
||||
status: "in_progress",
|
||||
originKind: "task_watchdog",
|
||||
originId: sourceId,
|
||||
originFingerprint: `task_watchdog:${companyId}:${sourceId}`,
|
||||
});
|
||||
const nestedChildId = await seedIssue(companyId, {
|
||||
parentId: nestedWatchdogIssueId,
|
||||
status: "in_progress",
|
||||
});
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
await db.insert(heartbeatRuns).values({
|
||||
companyId,
|
||||
agentId,
|
||||
status: "running",
|
||||
invocationSource: "assignment",
|
||||
contextSnapshot: { issueId: nestedChildId },
|
||||
});
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 1, live: 0 });
|
||||
expect(wakes).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("reconciles ancestor watchdogs for a descendant issue mutation", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-ANCESTOR", status: "done" });
|
||||
const childId = await seedIssue(companyId, { parentId: sourceId, status: "done" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileForIssueAndAncestors(companyId, childId);
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 1 });
|
||||
expect(wakes).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("marks a completed watchdog fingerprint reviewed, then reuses the same issue for a later stopped state", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-3", status: "done" });
|
||||
const childId = await seedIssue(companyId, { parentId: sourceId, status: "done" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
await service.reconcileTaskWatchdogs({ companyId });
|
||||
const [firstWatchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
const watchdogIssueId = firstWatchdog!.watchdogIssueId!;
|
||||
const [firstWatchdogIssue] = await db.select().from(issues).where(eq(issues.id, watchdogIssueId));
|
||||
expect(firstWatchdogIssue?.originFingerprint).toBe(firstWatchdog?.lastObservedFingerprint);
|
||||
await db.update(issues).set({ status: "done", updatedAt: new Date() }).where(eq(issues.id, watchdogIssueId));
|
||||
|
||||
const reviewed = await service.reconcileTaskWatchdogs({ companyId });
|
||||
expect(reviewed).toMatchObject({ checked: 1, triggered: 0, alreadyReviewed: 1 });
|
||||
const [reviewedWatchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
expect(reviewedWatchdog?.lastReviewedFingerprint).toBe(firstWatchdog?.lastObservedFingerprint);
|
||||
|
||||
await db
|
||||
.update(issues)
|
||||
.set({ status: "blocked", updatedAt: new Date(Date.now() + 60_000) })
|
||||
.where(eq(issues.id, childId));
|
||||
const retriggered = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(retriggered).toMatchObject({ checked: 1, triggered: 1 });
|
||||
const watchdogIssues = await db
|
||||
.select()
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "task_watchdog")));
|
||||
expect(watchdogIssues).toHaveLength(1);
|
||||
expect(watchdogIssues[0]).toMatchObject({ id: watchdogIssueId, status: "todo" });
|
||||
const comments = await db
|
||||
.select()
|
||||
.from(issueComments)
|
||||
.where(eq(issueComments.issueId, watchdogIssueId));
|
||||
expect(comments.some((comment) => comment.body.includes("Stopped fingerprint"))).toBe(true);
|
||||
expect(wakes.length).toBe(2);
|
||||
});
|
||||
|
||||
it("does not let an old terminal watchdog review mark a newer observed fingerprint reviewed", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-STALE", status: "done" });
|
||||
const childId = await seedIssue(companyId, { parentId: sourceId, status: "done" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
await service.reconcileTaskWatchdogs({ companyId });
|
||||
const [firstWatchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
const oldFingerprint = firstWatchdog!.lastObservedFingerprint!;
|
||||
const watchdogIssueId = firstWatchdog!.watchdogIssueId!;
|
||||
const watchdogRunId = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({
|
||||
id: watchdogRunId,
|
||||
companyId,
|
||||
agentId,
|
||||
status: "running",
|
||||
invocationSource: "assignment",
|
||||
contextSnapshot: { issueId: watchdogIssueId },
|
||||
});
|
||||
|
||||
await db
|
||||
.update(issues)
|
||||
.set({ status: "blocked", updatedAt: new Date(Date.now() + 60_000) })
|
||||
.where(eq(issues.id, childId));
|
||||
const changedWhileReviewLive = await service.reconcileTaskWatchdogs({ companyId });
|
||||
expect(changedWhileReviewLive).toMatchObject({ checked: 1, triggered: 0, live: 1 });
|
||||
|
||||
const [observedWhileLive] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
const newerFingerprint = observedWhileLive!.lastObservedFingerprint!;
|
||||
expect(newerFingerprint).not.toBe(oldFingerprint);
|
||||
const [stillBoundReview] = await db.select().from(issues).where(eq(issues.id, watchdogIssueId));
|
||||
expect(stillBoundReview?.originFingerprint).toBe(oldFingerprint);
|
||||
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, watchdogRunId));
|
||||
await db.update(issues).set({ status: "done", updatedAt: new Date() }).where(eq(issues.id, watchdogIssueId));
|
||||
const afterOldReviewCompletes = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(afterOldReviewCompletes).toMatchObject({ checked: 1, triggered: 1 });
|
||||
const [reviewedWatchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
expect(reviewedWatchdog?.lastReviewedFingerprint).toBe(oldFingerprint);
|
||||
expect(reviewedWatchdog?.lastReviewedFingerprint).not.toBe(newerFingerprint);
|
||||
const [reopenedWatchdogIssue] = await db.select().from(issues).where(eq(issues.id, watchdogIssueId));
|
||||
expect(reopenedWatchdogIssue).toMatchObject({
|
||||
status: "todo",
|
||||
originFingerprint: newerFingerprint,
|
||||
});
|
||||
const reviewActivities = await db
|
||||
.select()
|
||||
.from(activityLog)
|
||||
.where(and(eq(activityLog.entityId, sourceId), eq(activityLog.action, "issue.task_watchdog_fingerprint_reviewed")));
|
||||
expect(reviewActivities).toHaveLength(1);
|
||||
expect(reviewActivities[0]?.details).toMatchObject({
|
||||
reviewedFingerprint: oldFingerprint,
|
||||
lastObservedFingerprint: newerFingerprint,
|
||||
});
|
||||
expect(wakes.length).toBe(2);
|
||||
});
|
||||
|
||||
it("revalidates stale watchdog reviews against current source evidence before allowing mutations", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-REVALIDATE", status: "blocked" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service } = createService();
|
||||
|
||||
await service.reconcileTaskWatchdogs({ companyId });
|
||||
const [watchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
const originalFingerprint = watchdog!.lastObservedFingerprint!;
|
||||
expect(originalFingerprint).toMatch(/^task_watchdog_stop:/);
|
||||
|
||||
const later = new Date(Date.now() + 60_000);
|
||||
await db.insert(issueComments).values({
|
||||
companyId,
|
||||
issueId: sourceId,
|
||||
authorType: "agent",
|
||||
body: "Fresh source evidence.",
|
||||
updatedAt: later,
|
||||
createdAt: later,
|
||||
});
|
||||
await seedIssueDocument(companyId, sourceId, new Date(later.getTime() + 1_000));
|
||||
await seedIssueWorkProduct(companyId, sourceId, new Date(later.getTime() + 2_000));
|
||||
|
||||
const revalidated = await service.revalidateMutationScope({
|
||||
kind: "watchdog",
|
||||
watchdogId: watchdog!.id,
|
||||
companyId,
|
||||
watchedIssueId: sourceId,
|
||||
stopFingerprint: originalFingerprint,
|
||||
});
|
||||
|
||||
expect(revalidated.allowed).toBe(false);
|
||||
expect(revalidated.reason).toContain("stop fingerprint changed");
|
||||
expect(revalidated.classification?.state).toBe("stopped");
|
||||
if (revalidated.classification?.state !== "stopped") throw new Error("Expected stopped classification");
|
||||
expect(revalidated.classification.stopFingerprint).not.toBe(originalFingerprint);
|
||||
expect(revalidated.classification.stoppedLeaves[0]).toMatchObject({
|
||||
latestCommentAt: later.toISOString(),
|
||||
latestDocumentAt: new Date(later.getTime() + 1_000).toISOString(),
|
||||
latestWorkProductAt: new Date(later.getTime() + 2_000).toISOString(),
|
||||
});
|
||||
});
|
||||
|
||||
it("revalidates a stale watchdog review as live when the source gets a fresh run path", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-LIVE-REVALIDATE", status: "blocked" });
|
||||
const agentId = await seedAgent(companyId);
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service } = createService();
|
||||
|
||||
await service.reconcileTaskWatchdogs({ companyId });
|
||||
const [watchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
const originalFingerprint = watchdog!.lastObservedFingerprint!;
|
||||
await db.insert(heartbeatRuns).values({
|
||||
companyId,
|
||||
agentId,
|
||||
status: "running",
|
||||
invocationSource: "assignment",
|
||||
contextSnapshot: { issueId: sourceId },
|
||||
});
|
||||
|
||||
const revalidated = await service.revalidateMutationScope({
|
||||
kind: "watchdog",
|
||||
watchdogId: watchdog!.id,
|
||||
companyId,
|
||||
watchedIssueId: sourceId,
|
||||
stopFingerprint: originalFingerprint,
|
||||
});
|
||||
|
||||
expect(revalidated.allowed).toBe(false);
|
||||
expect(revalidated.reason).toContain("now has a live");
|
||||
expect(revalidated.classification?.state).toBe("live");
|
||||
});
|
||||
|
||||
it("does not raise a stopped-subtree review while a freshly-created assigned issue's first run is starting", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent(companyId);
|
||||
// Issue + watchdog created in the same flow; the assignment run row is not
|
||||
// yet visible to this evaluation (create-race).
|
||||
const sourceId = await seedIssue(companyId, {
|
||||
identifier: "WDOG-RACE",
|
||||
status: "todo",
|
||||
assigneeAgentId: agentId,
|
||||
createdAt: new Date(),
|
||||
});
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 0, pendingFirstRun: 1 });
|
||||
expect(wakes).toHaveLength(0);
|
||||
const watchdogIssues = await db
|
||||
.select({ id: issues.id })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "task_watchdog")));
|
||||
expect(watchdogIssues).toHaveLength(0);
|
||||
const [watchdog] = await db.select().from(issueWatchdogs).where(eq(issueWatchdogs.issueId, sourceId));
|
||||
expect(watchdog?.triggerCount).toBe(0);
|
||||
});
|
||||
|
||||
it("still triggers a genuinely idle assigned issue once it is past the first-run grace window", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent(companyId);
|
||||
// Established issue (default createdAt is an hour ago), assigned, non-terminal,
|
||||
// with no live run or queued wake.
|
||||
const sourceId = await seedIssue(companyId, {
|
||||
identifier: "WDOG-IDLE",
|
||||
status: "todo",
|
||||
assigneeAgentId: agentId,
|
||||
});
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 1 });
|
||||
expect(wakes).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("does not defer once the freshly-created issue has a terminal run on record", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent(companyId);
|
||||
const sourceId = await seedIssue(companyId, {
|
||||
identifier: "WDOG-RAN",
|
||||
status: "blocked",
|
||||
assigneeAgentId: agentId,
|
||||
createdAt: new Date(),
|
||||
});
|
||||
await seedWatchdog(companyId, sourceId, agentId);
|
||||
// A run for this issue already reached a terminal status, so the stop is
|
||||
// genuine even though the issue was just created.
|
||||
await db.insert(heartbeatRuns).values({
|
||||
companyId,
|
||||
agentId,
|
||||
status: "succeeded",
|
||||
invocationSource: "assignment",
|
||||
contextSnapshot: { issueId: sourceId },
|
||||
});
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 1 });
|
||||
expect(wakes).toHaveLength(1);
|
||||
});
|
||||
|
||||
it("does not recursively trigger a watchdog configured on a task-watchdog issue", async () => {
|
||||
const companyId = await seedCompany();
|
||||
const agentId = await seedAgent(companyId);
|
||||
const sourceId = await seedIssue(companyId, { identifier: "WDOG-4", status: "done" });
|
||||
const watchdogIssueId = await seedIssue(companyId, {
|
||||
parentId: sourceId,
|
||||
status: "done",
|
||||
originKind: "task_watchdog",
|
||||
originId: sourceId,
|
||||
originFingerprint: `task_watchdog:${companyId}:${sourceId}`,
|
||||
});
|
||||
await seedIssue(companyId, { parentId: watchdogIssueId, status: "done" });
|
||||
await seedWatchdog(companyId, watchdogIssueId, agentId);
|
||||
const { service, wakes } = createService();
|
||||
|
||||
const result = await service.reconcileTaskWatchdogs({ companyId });
|
||||
|
||||
expect(result).toMatchObject({ checked: 1, triggered: 0 });
|
||||
expect(wakes).toHaveLength(0);
|
||||
const watchdogIssues = await db
|
||||
.select()
|
||||
.from(issues)
|
||||
.where(and(eq(issues.companyId, companyId), eq(issues.originKind, "task_watchdog")));
|
||||
expect(watchdogIssues).toHaveLength(1);
|
||||
});
|
||||
});
|
||||
@@ -802,6 +802,14 @@ export async function startServer(): Promise<StartedServer> {
|
||||
);
|
||||
}
|
||||
|
||||
const taskWatchdogsReconciled = await heartbeat.reconcileTaskWatchdogs();
|
||||
if (taskWatchdogsReconciled.triggered > 0) {
|
||||
logger.warn(
|
||||
{ ...taskWatchdogsReconciled },
|
||||
"startup task-watchdog reconciliation triggered watchdog work",
|
||||
);
|
||||
}
|
||||
|
||||
const scanned = await heartbeat.scanSilentActiveRuns();
|
||||
if (scanned.created > 0 || scanned.escalated > 0) {
|
||||
logger.warn({ ...scanned }, "startup active-run output watchdog created review work");
|
||||
@@ -871,6 +879,12 @@ export async function startServer(): Promise<StartedServer> {
|
||||
logger.warn({ ...reconciled }, "periodic issue-graph liveness reconciliation created escalations");
|
||||
}
|
||||
})
|
||||
.then(async () => {
|
||||
const reconciled = await heartbeat.reconcileTaskWatchdogs();
|
||||
if (reconciled.triggered > 0) {
|
||||
logger.warn({ ...reconciled }, "periodic task-watchdog reconciliation triggered watchdog work");
|
||||
}
|
||||
})
|
||||
.then(async () => {
|
||||
const scanned = await heartbeat.scanSilentActiveRuns();
|
||||
if (scanned.created > 0 || scanned.escalated > 0) {
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { Router, type Request } from "express";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import {
|
||||
addApprovalCommentSchema,
|
||||
createApprovalSchema,
|
||||
@@ -28,6 +29,16 @@ function redactApprovalPayload<T extends { payload: Record<string, unknown> }>(a
|
||||
};
|
||||
}
|
||||
|
||||
function isStatusOnlyCheapRecoveryContext(contextSnapshot: unknown) {
|
||||
if (!contextSnapshot || typeof contextSnapshot !== "object" || Array.isArray(contextSnapshot)) return false;
|
||||
const context = contextSnapshot as Record<string, unknown>;
|
||||
return context.modelProfile === "cheap" &&
|
||||
context.recoveryIntent === "status_only" &&
|
||||
context.allowDeliverableWork === false &&
|
||||
context.allowDocumentUpdates === false &&
|
||||
context.resumeRequiresNormalModel === true;
|
||||
}
|
||||
|
||||
export function approvalRoutes(
|
||||
db: Db,
|
||||
options: { pluginWorkerManager?: PluginWorkerManager } = {},
|
||||
@@ -62,6 +73,37 @@ export function approvalRoutes(
|
||||
return false;
|
||||
}
|
||||
|
||||
async function assertApprovalMutationAllowedByRunContext(req: Request, res: any, companyId: string) {
|
||||
if (req.actor.type !== "agent") return true;
|
||||
const runId = req.actor.runId?.trim();
|
||||
if (!runId || !req.actor.agentId) return true;
|
||||
|
||||
const run = await db
|
||||
.select({
|
||||
id: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
agentId: heartbeatRuns.agentId,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, runId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!run || run.companyId !== companyId || run.agentId !== req.actor.agentId) return true;
|
||||
if (!isStatusOnlyCheapRecoveryContext(run.contextSnapshot)) return true;
|
||||
|
||||
res.status(403).json({
|
||||
error: "Cheap status-only recovery runs cannot create or modify approvals",
|
||||
details: {
|
||||
companyId,
|
||||
runId: run.id,
|
||||
modelProfile: "cheap",
|
||||
recoveryIntent: "status_only",
|
||||
resumeRequiresNormalModel: true,
|
||||
},
|
||||
});
|
||||
return false;
|
||||
}
|
||||
|
||||
router.get("/companies/:companyId/approvals", async (req, res) => {
|
||||
const companyId = req.params.companyId as string;
|
||||
assertCompanyAccess(req, companyId);
|
||||
@@ -87,6 +129,7 @@ export function approvalRoutes(
|
||||
const companyId = req.params.companyId as string;
|
||||
assertCompanyAccess(req, companyId);
|
||||
if (!(await assertApprovalAccessAllowed(req, res, companyId))) return;
|
||||
if (!(await assertApprovalMutationAllowedByRunContext(req, res, companyId))) return;
|
||||
const rawIssueIds = req.body.issueIds;
|
||||
const issueIds = Array.isArray(rawIssueIds)
|
||||
? rawIssueIds.filter((value: unknown): value is string => typeof value === "string")
|
||||
@@ -306,6 +349,7 @@ export function approvalRoutes(
|
||||
return;
|
||||
}
|
||||
assertCompanyAccess(req, existing.companyId);
|
||||
if (!(await assertApprovalMutationAllowedByRunContext(req, res, existing.companyId))) return;
|
||||
|
||||
if (req.actor.type === "agent" && req.actor.agentId !== existing.requestedByAgentId) {
|
||||
res.status(403).json({ error: "Only requesting agent can resubmit this approval" });
|
||||
@@ -356,6 +400,7 @@ export function approvalRoutes(
|
||||
return;
|
||||
}
|
||||
assertCompanyAccess(req, approval.companyId);
|
||||
if (!(await assertApprovalMutationAllowedByRunContext(req, res, approval.companyId))) return;
|
||||
const actor = getActorInfo(req);
|
||||
const comment = await svc.addComment(id, req.body.body, {
|
||||
agentId: actor.agentId ?? undefined,
|
||||
|
||||
@@ -64,7 +64,8 @@ export function instanceSettingsRoutes(db: Db) {
|
||||
|
||||
router.get("/instance/settings/experimental", async (req, res) => {
|
||||
// Experimental settings are readable by any authenticated org member
|
||||
// or instance admin. Only PATCH requires instance-admin.
|
||||
// or instance admin. Updating them remains instance-admin only because
|
||||
// this payload includes instance-wide operational controls.
|
||||
assertBoardOrgAccess(req);
|
||||
res.json(await svc.getExperimental());
|
||||
});
|
||||
|
||||
+722
-31
File diff suppressed because it is too large
Load Diff
@@ -26,6 +26,7 @@ import {
|
||||
upsertIssueDocumentSchema,
|
||||
restoreIssueDocumentRevisionSchema,
|
||||
upsertIssueFeedbackVoteSchema,
|
||||
upsertIssueWatchdogSchema,
|
||||
// Project
|
||||
createProjectSchema,
|
||||
updateProjectSchema,
|
||||
@@ -1380,6 +1381,36 @@ registry.registerPath({
|
||||
responses: { 200: r.ok(), 401: r.unauthorized },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "get",
|
||||
path: "/api/issues/{id}/watchdog",
|
||||
tags: ["issues"],
|
||||
summary: "Get active issue watchdog",
|
||||
request: { params: z.object({ id: z.string() }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 404: r.notFound },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "put",
|
||||
path: "/api/issues/{id}/watchdog",
|
||||
tags: ["issues"],
|
||||
summary: "Create or update an issue watchdog",
|
||||
request: {
|
||||
params: z.object({ id: z.string() }),
|
||||
body: jsonBody(upsertIssueWatchdogSchema),
|
||||
},
|
||||
responses: { 200: r.ok(), 400: r.badRequest, 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "delete",
|
||||
path: "/api/issues/{id}/watchdog",
|
||||
tags: ["issues"],
|
||||
summary: "Disable an issue watchdog",
|
||||
request: { params: z.object({ id: z.string() }) },
|
||||
responses: { 200: r.ok(), 401: r.unauthorized, 403: r.forbidden, 404: r.notFound },
|
||||
});
|
||||
|
||||
registry.registerPath({
|
||||
method: "get",
|
||||
path: "/api/issues/{id}/work-products",
|
||||
|
||||
@@ -158,6 +158,7 @@ import {
|
||||
} from "./recovery/model-profile-hint.js";
|
||||
import { recoveryService } from "./recovery/service.js";
|
||||
import { productivityReviewService } from "./productivity-review.js";
|
||||
import { taskWatchdogService } from "./task-watchdogs.js";
|
||||
import { withAgentStartLock } from "./agent-start-lock.js";
|
||||
import {
|
||||
evaluateAgentInvokability,
|
||||
@@ -2792,6 +2793,7 @@ export async function buildPaperclipWakePayload(input: {
|
||||
? input.contextSnapshot.unresolvedBlockerSummaries
|
||||
: [],
|
||||
executionStage: Object.keys(executionStage).length > 0 ? executionStage : null,
|
||||
taskWatchdog: (input.contextSnapshot.taskWatchdog ?? null) as unknown,
|
||||
continuationSummary: safeContinuationSummary
|
||||
? {
|
||||
key: safeContinuationSummary.key,
|
||||
@@ -3219,6 +3221,7 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
const budgets = budgetService(db, budgetHooks);
|
||||
const recovery = recoveryService(db, { enqueueWakeup });
|
||||
const productivityReviews = productivityReviewService(db, { enqueueWakeup });
|
||||
const taskWatchdogs = taskWatchdogService(db, { enqueueWakeup });
|
||||
let unsafeTextProjectionPromise: Promise<boolean> | null = null;
|
||||
|
||||
async function releaseEnvironmentLeasesForRun(input: {
|
||||
@@ -7744,6 +7747,10 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
return productivityReviews.reconcileProductivityReviews(opts);
|
||||
}
|
||||
|
||||
async function reconcileTaskWatchdogs(opts?: { companyId?: string | null; runId?: string | null }) {
|
||||
return taskWatchdogs.reconcileTaskWatchdogs(opts);
|
||||
}
|
||||
|
||||
async function buildRunOutputSilence(
|
||||
run: Pick<
|
||||
typeof heartbeatRuns.$inferSelect,
|
||||
@@ -11738,6 +11745,8 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
|
||||
|
||||
reconcileProductivityReviews,
|
||||
|
||||
reconcileTaskWatchdogs,
|
||||
|
||||
buildRunOutputSilence,
|
||||
|
||||
tickTimers: async (now = new Date()) => {
|
||||
|
||||
@@ -27,6 +27,12 @@ export { issueTreeControlService } from "./issue-tree-control.js";
|
||||
export { issueApprovalService } from "./issue-approvals.js";
|
||||
export { issueReferenceService } from "./issue-references.js";
|
||||
export { issueRecoveryActionService } from "./issue-recovery-actions.js";
|
||||
export { taskWatchdogService } from "./task-watchdogs.js";
|
||||
export {
|
||||
issueIsInTaskWatchdogSubtree,
|
||||
resolveTaskWatchdogMutationScope,
|
||||
taskWatchdogScopeAllowsIssueMutation,
|
||||
} from "./task-watchdog-scope.js";
|
||||
export { goalService } from "./goals.js";
|
||||
export { activityService, type ActivityFilters } from "./activity.js";
|
||||
export { approvalService } from "./approvals.js";
|
||||
|
||||
@@ -49,6 +49,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta
|
||||
enableConferenceRoomChat: parsed.data.enableConferenceRoomChat ?? false,
|
||||
enableIssuePlanDecompositions: parsed.data.enableIssuePlanDecompositions ?? false,
|
||||
enableExperimentalFileViewer: parsed.data.enableExperimentalFileViewer ?? false,
|
||||
enableTaskWatchdogs: parsed.data.enableTaskWatchdogs ?? false,
|
||||
enableCloudSync: parsed.data.enableCloudSync ?? false,
|
||||
autoRestartDevServerWhenIdle: parsed.data.autoRestartDevServerWhenIdle ?? false,
|
||||
enableIssueGraphLivenessAutoRecovery: parsed.data.enableIssueGraphLivenessAutoRecovery ?? false,
|
||||
@@ -64,6 +65,7 @@ export function normalizeExperimentalSettings(raw: unknown): InstanceExperimenta
|
||||
enableConferenceRoomChat: false,
|
||||
enableIssuePlanDecompositions: false,
|
||||
enableExperimentalFileViewer: false,
|
||||
enableTaskWatchdogs: false,
|
||||
enableCloudSync: false,
|
||||
autoRestartDevServerWhenIdle: false,
|
||||
enableIssueGraphLivenessAutoRecovery: false,
|
||||
|
||||
@@ -19,6 +19,7 @@ import {
|
||||
issueAttachments,
|
||||
issueInboxArchives,
|
||||
issueLabels,
|
||||
issueWatchdogs,
|
||||
issuePlanDecompositions,
|
||||
issueRecoveryActions,
|
||||
issueRelations,
|
||||
@@ -44,6 +45,7 @@ import type {
|
||||
IssueProductivityReview,
|
||||
IssueProductivityReviewTrigger,
|
||||
IssueRelationIssueSummary,
|
||||
IssueWatchdogSummary,
|
||||
LowTrustBoundary,
|
||||
SuccessfulRunHandoffState,
|
||||
} from "@paperclipai/shared";
|
||||
@@ -76,6 +78,10 @@ import { resolveIssueGoalId, resolveNextIssueGoalId } from "./issue-goal-fallbac
|
||||
import { getRunLogStore } from "./run-log-store.js";
|
||||
import { getDefaultCompanyGoal } from "./goals.js";
|
||||
import { assertAssignableAgent } from "./agent-assignability.js";
|
||||
import {
|
||||
summarizeIssueWatchdog,
|
||||
upsertIssueWatchdogForIssue,
|
||||
} from "./task-watchdogs.js";
|
||||
import {
|
||||
isVerifiedIssueTreeControlInteractionWake,
|
||||
issueTreeControlService,
|
||||
@@ -302,7 +308,11 @@ type IssueScheduledRetryRow = {
|
||||
error?: string | null;
|
||||
errorCode?: string | null;
|
||||
};
|
||||
type IssueWithLabels = IssueRow & { labels: IssueLabelRow[]; labelIds: string[] };
|
||||
type IssueWithLabels = IssueRow & {
|
||||
labels: IssueLabelRow[];
|
||||
labelIds: string[];
|
||||
watchdog?: IssueWatchdogSummary | null;
|
||||
};
|
||||
type IssueWithLabelsAndRun = IssueWithLabels & { activeRun: IssueActiveRunRow | null };
|
||||
type IssueUserCommentStats = {
|
||||
issueId: string;
|
||||
@@ -354,6 +364,8 @@ type IssueCreateInput = Omit<typeof issues.$inferInsert, "companyId"> & {
|
||||
labelIds?: string[];
|
||||
blockedByIssueIds?: string[];
|
||||
inheritExecutionWorkspaceFromIssueId?: string | null;
|
||||
watchdog?: { agentId: string; instructions?: string | null } | null;
|
||||
watchdogActorRunId?: string | null;
|
||||
};
|
||||
type IssueChildCreateInput = IssueCreateInput & {
|
||||
acceptanceCriteria?: string[];
|
||||
@@ -1143,17 +1155,49 @@ async function labelMapForIssues(dbOrTx: any, issueIds: string[]): Promise<Map<s
|
||||
|
||||
async function withIssueLabels(dbOrTx: any, rows: IssueRow[]): Promise<IssueWithLabels[]> {
|
||||
if (rows.length === 0) return [];
|
||||
const labelsByIssueId = await labelMapForIssues(dbOrTx, rows.map((row) => row.id));
|
||||
const issueIds = rows.map((row) => row.id);
|
||||
const [labelsByIssueId, watchdogByIssueId] = await Promise.all([
|
||||
labelMapForIssues(dbOrTx, issueIds),
|
||||
watchdogMapForIssues(dbOrTx, rows),
|
||||
]);
|
||||
return rows.map((row) => {
|
||||
const issueLabels = labelsByIssueId.get(row.id) ?? [];
|
||||
return {
|
||||
...row,
|
||||
labels: issueLabels,
|
||||
labelIds: issueLabels.map((label) => label.id),
|
||||
watchdog: watchdogByIssueId.get(row.id) ?? null,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
async function watchdogMapForIssues(dbOrTx: any, rows: IssueRow[]): Promise<Map<string, IssueWatchdogSummary>> {
|
||||
const map = new Map<string, IssueWatchdogSummary>();
|
||||
if (rows.length === 0) return map;
|
||||
const byCompany = new Map<string, string[]>();
|
||||
for (const row of rows) {
|
||||
const ids = byCompany.get(row.companyId) ?? [];
|
||||
ids.push(row.id);
|
||||
byCompany.set(row.companyId, ids);
|
||||
}
|
||||
for (const [companyId, issueIds] of byCompany.entries()) {
|
||||
for (const issueIdChunk of chunkList([...new Set(issueIds)], ISSUE_LIST_RELATED_QUERY_CHUNK_SIZE)) {
|
||||
const watchdogRows = await dbOrTx
|
||||
.select()
|
||||
.from(issueWatchdogs)
|
||||
.where(and(
|
||||
eq(issueWatchdogs.companyId, companyId),
|
||||
inArray(issueWatchdogs.issueId, issueIdChunk),
|
||||
eq(issueWatchdogs.status, "active"),
|
||||
));
|
||||
for (const row of watchdogRows) {
|
||||
map.set(row.issueId, summarizeIssueWatchdog(row));
|
||||
}
|
||||
}
|
||||
}
|
||||
return map;
|
||||
}
|
||||
|
||||
const ACTIVE_RUN_STATUSES = ["queued", "running"];
|
||||
const BLOCKER_ATTENTION_ACTIVE_RUN_STATUSES = ["queued", "running"];
|
||||
const BLOCKER_ATTENTION_ACTIVE_WAKE_STATUSES = ["queued", "deferred_issue_execution"];
|
||||
@@ -4857,6 +4901,8 @@ export function issueService(db: Db) {
|
||||
labelIds: inputLabelIds,
|
||||
blockedByIssueIds,
|
||||
inheritExecutionWorkspaceFromIssueId,
|
||||
watchdog,
|
||||
watchdogActorRunId,
|
||||
...issueData
|
||||
} = data;
|
||||
const isolatedWorkspacesEnabled = (await instanceSettings.getExperimental()).enableIsolatedWorkspaces;
|
||||
@@ -5073,6 +5119,17 @@ export function issueService(db: Db) {
|
||||
);
|
||||
|
||||
const [issue] = await tx.insert(issues).values(values).returning();
|
||||
if (watchdog) {
|
||||
await upsertIssueWatchdogForIssue(tx, companyId, issue.id, {
|
||||
agentId: watchdog.agentId,
|
||||
instructions: watchdog.instructions,
|
||||
actor: {
|
||||
agentId: issueData.createdByAgentId ?? null,
|
||||
userId: issueData.createdByUserId ?? null,
|
||||
runId: watchdogActorRunId ?? null,
|
||||
},
|
||||
});
|
||||
}
|
||||
if (inputLabelIds) {
|
||||
await syncIssueLabels(issue.id, companyId, inputLabelIds, tx);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
import { and, eq } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { heartbeatRuns, issues, issueWatchdogs } from "@paperclipai/db";
|
||||
|
||||
const MAX_WATCHDOG_SCOPE_ANCESTRY_DEPTH = 100;
|
||||
export const TASK_WATCHDOG_ORIGIN_KIND = "task_watchdog";
|
||||
|
||||
type AgentRunActor = {
|
||||
type: string;
|
||||
agentId?: string | null;
|
||||
companyId?: string | null;
|
||||
runId?: string | null;
|
||||
};
|
||||
|
||||
type IssueScopeTarget = {
|
||||
id: string;
|
||||
companyId: string;
|
||||
parentId?: string | null;
|
||||
};
|
||||
|
||||
export type TaskWatchdogMutationScope =
|
||||
| { kind: "none" }
|
||||
| { kind: "invalid"; detail: string }
|
||||
| {
|
||||
kind: "watchdog";
|
||||
watchdogId: string;
|
||||
companyId: string;
|
||||
watchedIssueId: string;
|
||||
watchdogIssueId: string | null;
|
||||
stopFingerprint: string | null;
|
||||
};
|
||||
|
||||
function isPlainRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === "object" && value !== null && !Array.isArray(value);
|
||||
}
|
||||
|
||||
function readString(value: unknown): string | null {
|
||||
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
|
||||
}
|
||||
|
||||
function readTaskWatchdogContext(contextSnapshot: unknown) {
|
||||
const context = isPlainRecord(contextSnapshot) ? contextSnapshot : null;
|
||||
const taskWatchdog = isPlainRecord(context?.taskWatchdog) ? context.taskWatchdog : null;
|
||||
if (!taskWatchdog && context?.taskWatchdog !== true) return null;
|
||||
return {
|
||||
watchedIssueId: readString(taskWatchdog?.watchedIssueId) ?? readString(context?.watchedIssueId),
|
||||
stopFingerprint: readString(taskWatchdog?.stopFingerprint) ?? readString(context?.stopFingerprint),
|
||||
};
|
||||
}
|
||||
|
||||
export async function resolveTaskWatchdogMutationScope(
|
||||
db: Db,
|
||||
actor: AgentRunActor,
|
||||
): Promise<TaskWatchdogMutationScope> {
|
||||
if (actor.type !== "agent") return { kind: "none" };
|
||||
const agentId = readString(actor.agentId);
|
||||
const runId = readString(actor.runId);
|
||||
const actorCompanyId = readString(actor.companyId);
|
||||
if (!agentId || !runId) return { kind: "none" };
|
||||
|
||||
const run = await db
|
||||
.select({
|
||||
id: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
agentId: heartbeatRuns.agentId,
|
||||
contextSnapshot: heartbeatRuns.contextSnapshot,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, runId))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
if (!run) return { kind: "none" };
|
||||
const taskWatchdog = readTaskWatchdogContext(run.contextSnapshot);
|
||||
if (!taskWatchdog) return { kind: "none" };
|
||||
if (run.agentId !== agentId || (actorCompanyId && run.companyId !== actorCompanyId)) {
|
||||
return {
|
||||
kind: "invalid",
|
||||
detail: "Task-watchdog run context does not belong to this agent.",
|
||||
};
|
||||
}
|
||||
|
||||
if (!taskWatchdog.watchedIssueId) {
|
||||
return {
|
||||
kind: "invalid",
|
||||
detail: "Task-watchdog run context is missing a persisted watched issue id.",
|
||||
};
|
||||
}
|
||||
|
||||
const watchdog = await db
|
||||
.select({
|
||||
id: issueWatchdogs.id,
|
||||
companyId: issueWatchdogs.companyId,
|
||||
issueId: issueWatchdogs.issueId,
|
||||
watchdogAgentId: issueWatchdogs.watchdogAgentId,
|
||||
watchdogIssueId: issueWatchdogs.watchdogIssueId,
|
||||
status: issueWatchdogs.status,
|
||||
})
|
||||
.from(issueWatchdogs)
|
||||
.where(and(
|
||||
eq(issueWatchdogs.companyId, run.companyId),
|
||||
eq(issueWatchdogs.issueId, taskWatchdog.watchedIssueId),
|
||||
eq(issueWatchdogs.watchdogAgentId, agentId),
|
||||
eq(issueWatchdogs.status, "active"),
|
||||
))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
|
||||
if (!watchdog) {
|
||||
return {
|
||||
kind: "invalid",
|
||||
detail: "Task-watchdog run context is not backed by an active persisted watchdog.",
|
||||
};
|
||||
}
|
||||
|
||||
return {
|
||||
kind: "watchdog",
|
||||
watchdogId: watchdog.id,
|
||||
companyId: watchdog.companyId,
|
||||
watchedIssueId: watchdog.issueId,
|
||||
watchdogIssueId: watchdog.watchdogIssueId ?? null,
|
||||
stopFingerprint: taskWatchdog.stopFingerprint,
|
||||
};
|
||||
}
|
||||
|
||||
export async function issueIsInTaskWatchdogSubtree(
|
||||
db: Db,
|
||||
companyId: string,
|
||||
issueId: string,
|
||||
watchedIssueId: string,
|
||||
) {
|
||||
let currentId: string | null = issueId;
|
||||
const seen = new Set<string>();
|
||||
|
||||
for (let depth = 0; currentId && depth < MAX_WATCHDOG_SCOPE_ANCESTRY_DEPTH; depth += 1) {
|
||||
if (seen.has(currentId)) return false;
|
||||
seen.add(currentId);
|
||||
|
||||
const parent: { id: string; companyId: string; parentId: string | null; originKind: string | null } | null = await db
|
||||
.select({ id: issues.id, companyId: issues.companyId, parentId: issues.parentId, originKind: issues.originKind })
|
||||
.from(issues)
|
||||
.where(and(eq(issues.id, currentId), eq(issues.companyId, companyId)))
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!parent) return false;
|
||||
if (parent.originKind === TASK_WATCHDOG_ORIGIN_KIND) return false;
|
||||
if (currentId === watchedIssueId) return true;
|
||||
currentId = parent.parentId ?? null;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
export async function taskWatchdogScopeAllowsIssueMutation(
|
||||
db: Db,
|
||||
scope: TaskWatchdogMutationScope,
|
||||
issue: IssueScopeTarget,
|
||||
opts: { allowWatchdogIssue?: boolean } = {},
|
||||
) {
|
||||
if (scope.kind !== "watchdog") return scope;
|
||||
if (issue.companyId !== scope.companyId) {
|
||||
return {
|
||||
kind: "invalid" as const,
|
||||
detail: "Task-watchdog mutation target is outside the watchdog company.",
|
||||
};
|
||||
}
|
||||
if (opts.allowWatchdogIssue !== false && scope.watchdogIssueId && issue.id === scope.watchdogIssueId) {
|
||||
return scope;
|
||||
}
|
||||
if (await issueIsInTaskWatchdogSubtree(db, scope.companyId, issue.id, scope.watchedIssueId)) {
|
||||
return scope;
|
||||
}
|
||||
return {
|
||||
kind: "invalid" as const,
|
||||
detail: "Task-watchdog runs can only mutate the watched issue subtree.",
|
||||
};
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user