Add heartbeat preflight budget caps (#8347)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Heartbeats are the control-plane path that turns scheduled,
comment-driven, or on-demand wakeups into adapter executions.
> - Budgeting and recurring work need enforcement before an adapter
starts, not only after model usage is recorded.
> - Empty timer wakes also need an opt-in fast-exit path so operators
can keep routine schedules without paying for no-op model turns.
> - This pull request adds heartbeat preflight gates for daily run and
daily cost caps, plus an explicit timer no-work skip policy.
> - The benefit is safer autonomous operation: capped agents stop before
new execution, queued work is cancelled cleanly at claim time, and
proactive agents still run by default unless the operator opts into
no-work skipping.

## Linked Issues or Issue Description

No public issue exists for this change. Inline bug report:

### What happened?

Heartbeat execution can start without enforcing per-agent daily
invocation and spend limits at the heartbeat boundary. A run that was
queued before a cap was reached can also be claimed later and invoke the
adapter unless the cap is checked again immediately before execution.
Operators also do not have an explicit opt-in fast-exit policy for
generic timer wakes with no actionable assigned work.

### Expected behavior

Configured daily run and daily cost caps should stop new heartbeat runs
before adapter execution. Already queued runs should be rechecked at
claim time and cancelled cleanly when a cap is now reached. Queued issue
runs cancelled by daily caps should release their issue execution locks
and promote deferred wakeups without entering immediate recovery loops.
Generic timer no-work skipping should be opt-in so proactive agents
continue to run by default.

### Steps to reproduce

1. Configure an agent heartbeat policy with a one-run daily cap or a
daily cost cap.
2. Create or queue heartbeat wakeups for that agent after the cap has
already been consumed.
3. Observe that without preflight and claim-time checks, the heartbeat
path can still enqueue or claim work that should be blocked before
adapter execution.

### Paperclip version or commit

Reproduced against `master` before this branch.

### Deployment mode

Local dev (`pnpm dev`) / built from source.

### Installation method

Built from source (`pnpm dev` / `pnpm build`).

### Agent adapter(s) involved

Not adapter-specific (core heartbeat scheduling and claim logic).

### Database mode

External Postgres in tests via embedded test harness.

### Access context

Not applicable.

### Node.js version

Node 20 in CI-compatible local development.

### Operating system

macOS local development, Linux CI-compatible tests.

### Relevant logs or output

The regression suite added in this PR covers the failing paths:

```shell
pnpm exec vitest run server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts
```

### Relevant config (if applicable)

```json
{
  "heartbeat": {
    "maxDailyRuns": 1,
    "maxDailyCostCents": 1,
    "skipTimerWhenNoActionableWork": true
  }
}
```

### Additional context

This affects recurring/autonomous operation because the safest place to
stop excess work is before adapter execution starts.

### Privacy checklist

Reviewed for sensitive data; no private logs, credentials, or local
instance URLs are included.

## What Changed

- Added heartbeat policy parsing for per-agent daily run caps, daily
cost caps, and opt-in no-actionable-work timer skipping.
- Added pre-queue daily cap checks while preserving same-issue wake
coalescing.
- Added claim-time cap checks so already queued runs are cancelled
before adapter execution when a cap is reached.
- Added skipped wakeup metadata for cap and timer fast-exit decisions.
- Released issue execution locks for queued issue runs cancelled by
daily caps, with deferred wake promotion and without immediate recovery
loops while caps are active.
- Added regression coverage for timer skipping, proactive default
behavior, run caps, cost caps, queued-run cancellation, started
cancelled runs, and deferred issue wake promotion.

## Verification

- `git diff --check`
- `node -c server/src/services/heartbeat.ts`
- `pnpm exec vitest run
server/src/__tests__/heartbeat-stale-queue-invalidation.test.ts`
- `pnpm --filter @paperclipai/server typecheck`
- Local autoreview: `skills/autoreview/scripts/autoreview --mode branch
--base origin/master --engine codex --model gpt-5.5 --thinking high`
  - Result: clean, no accepted/actionable findings

## Risks

- Medium operational risk because this changes heartbeat scheduling and
claim-time behavior.
- The no-actionable-work timer fast-exit is explicitly opt-in to avoid
suppressing proactive agents unexpectedly.
- Daily run caps count runs by `startedAt` so old queued rows do not
consume today’s cap, while started runs still count even if they later
end as cancelled.
- Queued issue-run cap cancellation uses the existing release/promotion
path with immediate recovery suppressed to avoid retry loops while caps
are active.

## Model Used

Codex with GPT-5.5 high reasoning assisted with implementation, local
testing, and autoreview. The final review gate used local autoreview
with `gpt-5.5` high reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [ ] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
This commit is contained in:
levineam
2026-06-20 00:39:50 -04:00
committed by GitHub
parent 323b6220cc
commit 631b7806ed
2 changed files with 1020 additions and 40 deletions
@@ -5,6 +5,7 @@ import {
agents,
agentWakeupRequests,
companies,
costEvents,
createDb,
documentRevisions,
documents,
@@ -96,6 +97,7 @@ async function cleanupHeartbeatInvalidationFixture(db: ReturnType<typeof createD
"issue_tree_holds",
"issues",
"heartbeat_run_events",
"cost_events",
"activity_log",
"heartbeat_runs",
"agent_wakeup_requests",
@@ -124,6 +126,7 @@ type SeedOptions = {
agentName?: string;
agentRole?: string;
maxConcurrentRuns?: number;
heartbeatConfig?: Record<string, unknown>;
};
type SeedResult = {
@@ -201,6 +204,7 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => {
heartbeat: {
wakeOnDemand: true,
maxConcurrentRuns: opts.maxConcurrentRuns ?? 1,
...(opts.heartbeatConfig ?? {}),
},
},
permissions: {},
@@ -288,6 +292,701 @@ describeEmbeddedPostgres("heartbeat stale queued-run invalidation", () => {
});
}
it("skips generic timer wakes with no actionable assigned work before adapter execution", async () => {
const { agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
enabled: true,
skipTimerWhenNoActionableWork: true,
},
});
const run = await heartbeat.wakeup(agentId, {
source: "timer",
triggerDetail: "schedule",
});
expect(run).toBeNull();
expect(mockAdapterExecute).not.toHaveBeenCalled();
const [wakeup] = await db
.select({
status: agentWakeupRequests.status,
reason: agentWakeupRequests.reason,
payload: agentWakeupRequests.payload,
})
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.agentId, agentId));
const runRows = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns);
expect(wakeup).toMatchObject({
status: "skipped",
reason: "heartbeat.timer.no_actionable_work",
});
expect(wakeup?.payload).toMatchObject({
heartbeatSkip: {
reason: expect.stringContaining("No assigned todo or in_progress issue"),
},
});
expect(runRows).toHaveLength(0);
});
it("rate-limits skipped generic timer wakes by advancing the timer baseline", async () => {
const { agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
enabled: true,
intervalSec: 60,
skipTimerWhenNoActionableWork: true,
},
});
const now = new Date();
await db
.update(agents)
.set({ lastHeartbeatAt: new Date(now.getTime() - 120_000) })
.where(eq(agents.id, agentId));
const firstTick = await heartbeat.tickTimers(now);
const secondTick = await heartbeat.tickTimers(now);
expect(firstTick.skipped).toBe(1);
expect(secondTick.skipped).toBe(0);
expect(mockAdapterExecute).not.toHaveBeenCalled();
const wakeups = await db
.select({ reason: agentWakeupRequests.reason })
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.agentId, agentId));
const [agent] = await db
.select({ lastHeartbeatAt: agents.lastHeartbeatAt })
.from(agents)
.where(eq(agents.id, agentId));
expect(wakeups).toHaveLength(1);
expect(wakeups[0]?.reason).toBe("heartbeat.timer.no_actionable_work");
expect(agent?.lastHeartbeatAt).toBeInstanceOf(Date);
expect(agent?.lastHeartbeatAt?.getTime()).toBeGreaterThan(now.getTime() - 120_000);
});
it("allows generic timer wakes when the agent has assigned todo work", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
enabled: true,
skipTimerWhenNoActionableWork: true,
},
});
await db.insert(issues).values({
id: randomUUID(),
companyId,
title: "Assigned work",
status: "todo",
priority: "high",
assigneeAgentId: agentId,
});
const run = await heartbeat.wakeup(agentId, {
source: "timer",
triggerDetail: "schedule",
});
expect(run).not.toBeNull();
await waitForCondition(async () => countExecuteCallsForRun(run!.id) > 0);
expect(countExecuteCallsForRun(run!.id)).toBe(1);
});
it("runs generic timer wakes by default for proactive agents without assigned issue work", async () => {
const { agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
enabled: true,
},
});
const run = await heartbeat.wakeup(agentId, {
source: "timer",
triggerDetail: "schedule",
});
expect(run).not.toBeNull();
await waitForCondition(async () => countExecuteCallsForRun(run!.id) > 0);
expect(countExecuteCallsForRun(run!.id)).toBe(1);
});
it("skips wakes before queueing when per-agent daily run cap is reached", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
maxDailyRuns: 1,
},
});
await db.insert(heartbeatRuns).values({
id: randomUUID(),
companyId,
agentId,
invocationSource: "on_demand",
triggerDetail: "manual",
status: "succeeded",
createdAt: new Date(),
startedAt: new Date(),
finishedAt: new Date(),
contextSnapshot: {},
});
const run = await heartbeat.wakeup(agentId, {
source: "on_demand",
triggerDetail: "manual",
});
expect(run).toBeNull();
expect(mockAdapterExecute).not.toHaveBeenCalled();
const [wakeup] = await db
.select({
status: agentWakeupRequests.status,
reason: agentWakeupRequests.reason,
payload: agentWakeupRequests.payload,
})
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.agentId, agentId));
expect(wakeup).toMatchObject({
status: "skipped",
reason: "heartbeat.daily_run_limit",
});
expect(wakeup?.payload).toMatchObject({
heartbeatSkip: {
observed: 1,
limit: 1,
},
});
});
it("treats zero daily run cap as a hard stop", async () => {
const { agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
maxDailyRuns: 0,
},
});
const run = await heartbeat.wakeup(agentId, {
source: "on_demand",
triggerDetail: "manual",
});
expect(run).toBeNull();
expect(mockAdapterExecute).not.toHaveBeenCalled();
const [wakeup] = await db
.select({
status: agentWakeupRequests.status,
reason: agentWakeupRequests.reason,
payload: agentWakeupRequests.payload,
})
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.agentId, agentId));
expect(wakeup).toMatchObject({
status: "skipped",
reason: "heartbeat.daily_run_limit",
});
expect(wakeup?.payload).toMatchObject({
heartbeatSkip: {
observed: 0,
limit: 0,
},
});
});
it("counts started cancelled runs toward the per-agent daily run cap", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
maxDailyRuns: 1,
},
});
await db.insert(heartbeatRuns).values({
id: randomUUID(),
companyId,
agentId,
invocationSource: "on_demand",
triggerDetail: "manual",
status: "cancelled",
createdAt: new Date(),
startedAt: new Date(),
finishedAt: new Date(),
contextSnapshot: {},
});
const run = await heartbeat.wakeup(agentId, {
source: "on_demand",
triggerDetail: "manual",
});
expect(run).toBeNull();
expect(mockAdapterExecute).not.toHaveBeenCalled();
const [wakeup] = await db
.select({
status: agentWakeupRequests.status,
reason: agentWakeupRequests.reason,
payload: agentWakeupRequests.payload,
})
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.agentId, agentId));
expect(wakeup).toMatchObject({
status: "skipped",
reason: "heartbeat.daily_run_limit",
});
expect(wakeup?.payload).toMatchObject({
heartbeatSkip: {
observed: 1,
limit: 1,
},
});
});
it("coalesces same-issue wakes before enforcing the daily run cap", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
maxDailyRuns: 1,
},
});
const issueId = randomUUID();
const wakeupRequestId = randomUUID();
const queuedRunId = randomUUID();
await db.insert(heartbeatRuns).values({
id: randomUUID(),
companyId,
agentId,
invocationSource: "on_demand",
triggerDetail: "manual",
status: "succeeded",
createdAt: new Date(),
startedAt: new Date(),
finishedAt: new Date(),
contextSnapshot: {},
});
await db.insert(agentWakeupRequests).values({
id: wakeupRequestId,
companyId,
agentId,
source: "on_demand",
triggerDetail: "manual",
reason: "manual",
payload: { issueId },
status: "queued",
});
await db.insert(heartbeatRuns).values({
id: queuedRunId,
companyId,
agentId,
invocationSource: "on_demand",
triggerDetail: "manual",
status: "queued",
wakeupRequestId,
contextSnapshot: { issueId },
});
await db.insert(issues).values({
id: issueId,
companyId,
title: "Queued issue work",
status: "in_progress",
priority: "high",
assigneeAgentId: agentId,
executionRunId: queuedRunId,
});
await db
.update(agentWakeupRequests)
.set({ runId: queuedRunId })
.where(eq(agentWakeupRequests.id, wakeupRequestId));
const run = await heartbeat.wakeup(agentId, {
source: "on_demand",
triggerDetail: "manual",
payload: { issueId },
});
expect(run?.id).toBe(queuedRunId);
expect(mockAdapterExecute).not.toHaveBeenCalled();
const wakeups = await db
.select({
status: agentWakeupRequests.status,
reason: agentWakeupRequests.reason,
runId: agentWakeupRequests.runId,
})
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.agentId, agentId));
expect(wakeups).toEqual(
expect.arrayContaining([
expect.objectContaining({
status: "coalesced",
reason: "issue_execution_same_name",
runId: queuedRunId,
}),
]),
);
});
it("skips wakes before queueing when per-agent daily cost cap is reached", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
maxDailyCostCents: 75,
},
});
await db.insert(costEvents).values({
companyId,
agentId,
provider: "test",
biller: "test",
billingType: "metered_api",
model: "test-model",
inputTokens: 100,
outputTokens: 50,
costCents: 75,
occurredAt: new Date(),
});
const run = await heartbeat.wakeup(agentId, {
source: "on_demand",
triggerDetail: "manual",
});
expect(run).toBeNull();
expect(mockAdapterExecute).not.toHaveBeenCalled();
const [wakeup] = await db
.select({
status: agentWakeupRequests.status,
reason: agentWakeupRequests.reason,
payload: agentWakeupRequests.payload,
})
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.agentId, agentId));
expect(wakeup).toMatchObject({
status: "skipped",
reason: "heartbeat.daily_cost_limit",
});
expect(wakeup?.payload).toMatchObject({
heartbeatSkip: {
observed: 75,
limit: 75,
},
});
});
it("treats zero daily cost cap as a hard stop", async () => {
const { agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
maxDailyCostCents: 0,
},
});
const run = await heartbeat.wakeup(agentId, {
source: "on_demand",
triggerDetail: "manual",
});
expect(run).toBeNull();
expect(mockAdapterExecute).not.toHaveBeenCalled();
const [wakeup] = await db
.select({
status: agentWakeupRequests.status,
reason: agentWakeupRequests.reason,
payload: agentWakeupRequests.payload,
})
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.agentId, agentId));
expect(wakeup).toMatchObject({
status: "skipped",
reason: "heartbeat.daily_cost_limit",
});
expect(wakeup?.payload).toMatchObject({
heartbeatSkip: {
observed: 0,
limit: 0,
},
});
});
it("skips already queued runs before adapter execution when the daily cost cap is reached", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
maxDailyCostCents: 75,
},
});
const wakeupRequestId = randomUUID();
const runId = randomUUID();
await db.insert(agentWakeupRequests).values({
id: wakeupRequestId,
companyId,
agentId,
source: "on_demand",
triggerDetail: "manual",
reason: "manual",
payload: {},
status: "queued",
});
await db.insert(heartbeatRuns).values({
id: runId,
companyId,
agentId,
invocationSource: "on_demand",
triggerDetail: "manual",
status: "queued",
wakeupRequestId,
contextSnapshot: {},
});
await db
.update(agentWakeupRequests)
.set({ runId })
.where(eq(agentWakeupRequests.id, wakeupRequestId));
await db.insert(costEvents).values({
companyId,
agentId,
provider: "test",
biller: "test",
billingType: "metered_api",
model: "test-model",
inputTokens: 100,
outputTokens: 50,
costCents: 75,
occurredAt: new Date(),
});
await heartbeat.resumeQueuedRuns();
expect(mockAdapterExecute).not.toHaveBeenCalled();
const [run] = await db
.select({
status: heartbeatRuns.status,
errorCode: heartbeatRuns.errorCode,
resultJson: heartbeatRuns.resultJson,
})
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, runId));
const [wakeup] = await db
.select({
status: agentWakeupRequests.status,
error: agentWakeupRequests.error,
})
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.id, wakeupRequestId));
expect(run).toMatchObject({
status: "cancelled",
errorCode: "heartbeat.daily_cost_limit",
});
expect(run?.resultJson).toMatchObject({
stopReason: "heartbeat.daily_cost_limit",
observed: 75,
limit: 75,
});
expect(wakeup).toMatchObject({
status: "skipped",
error: expect.stringContaining("per-day heartbeat budget cap"),
});
});
it("skips already queued issue runs at the daily run cap and releases the execution lock", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
maxDailyRuns: 1,
},
});
const issueId = randomUUID();
const wakeupRequestId = randomUUID();
const queuedRunId = randomUUID();
await db.insert(heartbeatRuns).values({
id: randomUUID(),
companyId,
agentId,
invocationSource: "on_demand",
triggerDetail: "manual",
status: "succeeded",
createdAt: new Date(),
startedAt: new Date(),
finishedAt: new Date(),
contextSnapshot: {},
});
await db.insert(agentWakeupRequests).values({
id: wakeupRequestId,
companyId,
agentId,
source: "on_demand",
triggerDetail: "manual",
reason: "manual",
payload: { issueId },
status: "queued",
});
await db.insert(heartbeatRuns).values({
id: queuedRunId,
companyId,
agentId,
invocationSource: "on_demand",
triggerDetail: "manual",
status: "queued",
wakeupRequestId,
contextSnapshot: { issueId },
});
await db.insert(issues).values({
id: issueId,
companyId,
title: "Queued issue work",
status: "in_progress",
priority: "high",
assigneeAgentId: agentId,
executionRunId: queuedRunId,
});
await db
.update(agentWakeupRequests)
.set({ runId: queuedRunId })
.where(eq(agentWakeupRequests.id, wakeupRequestId));
await heartbeat.resumeQueuedRuns();
expect(mockAdapterExecute).not.toHaveBeenCalled();
const [run] = await db
.select({
status: heartbeatRuns.status,
errorCode: heartbeatRuns.errorCode,
})
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, queuedRunId));
const [wakeup] = await db
.select({ status: agentWakeupRequests.status })
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.id, wakeupRequestId));
const [issue] = await db
.select({ executionRunId: issues.executionRunId })
.from(issues)
.where(eq(issues.id, issueId));
expect(run).toMatchObject({
status: "cancelled",
errorCode: "heartbeat.daily_run_limit",
});
expect(wakeup).toMatchObject({ status: "skipped" });
expect(issue?.executionRunId).toBeNull();
});
it("promotes deferred issue wakes when a queued holder is cancelled by the daily run cap", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({
heartbeatConfig: {
maxDailyRuns: 1,
},
});
const peerAgentId = randomUUID();
const issueId = randomUUID();
const wakeupRequestId = randomUUID();
const queuedRunId = randomUUID();
const deferredWakeupId = randomUUID();
await db.insert(agents).values({
id: peerAgentId,
companyId,
name: "PeerAgent",
role: "engineer",
status: "active",
adapterType: "codex_local",
adapterConfig: {},
runtimeConfig: {
heartbeat: {
wakeOnDemand: true,
maxConcurrentRuns: 1,
},
},
permissions: {},
});
await db.insert(heartbeatRuns).values({
id: randomUUID(),
companyId,
agentId,
invocationSource: "on_demand",
triggerDetail: "manual",
status: "succeeded",
createdAt: new Date(),
startedAt: new Date(),
finishedAt: new Date(),
contextSnapshot: {},
});
await db.insert(agentWakeupRequests).values({
id: wakeupRequestId,
companyId,
agentId,
source: "on_demand",
triggerDetail: "manual",
reason: "manual",
payload: { issueId },
status: "queued",
});
await db.insert(heartbeatRuns).values({
id: queuedRunId,
companyId,
agentId,
invocationSource: "on_demand",
triggerDetail: "manual",
status: "queued",
wakeupRequestId,
contextSnapshot: { issueId },
});
await db.insert(issues).values({
id: issueId,
companyId,
title: "Queued issue work",
status: "in_progress",
priority: "high",
assigneeAgentId: agentId,
executionRunId: queuedRunId,
});
await db.insert(agentWakeupRequests).values({
id: deferredWakeupId,
companyId,
agentId: peerAgentId,
source: "comment",
triggerDetail: "mention",
reason: "issue_execution_deferred",
payload: {
issueId,
_paperclipWakeContext: {
issueId,
wakeReason: "issue_mention",
},
},
status: "deferred_issue_execution",
});
await db
.update(agentWakeupRequests)
.set({ runId: queuedRunId })
.where(eq(agentWakeupRequests.id, wakeupRequestId));
await heartbeat.resumeQueuedRuns();
await waitForCondition(async () => {
const [deferred] = await db
.select({ status: agentWakeupRequests.status, runId: agentWakeupRequests.runId })
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.id, deferredWakeupId));
return Boolean(deferred?.runId) && deferred?.status !== "deferred_issue_execution";
});
const [deferred] = await db
.select({ status: agentWakeupRequests.status, runId: agentWakeupRequests.runId })
.from(agentWakeupRequests)
.where(eq(agentWakeupRequests.id, deferredWakeupId));
const [promotedRun] = deferred?.runId
? await db
.select({ agentId: heartbeatRuns.agentId })
.from(heartbeatRuns)
.where(eq(heartbeatRuns.id, deferred.runId))
: [];
expect(deferred?.status).not.toBe("deferred_issue_execution");
expect(promotedRun?.agentId).toBe(peerAgentId);
});
it("cancels queued runs when the issue assignee changes before the run starts", async () => {
const { companyId, agentId } = await seedCompanyAndAgent({ agentName: "OriginalCoder" });
const replacementAgentId = randomUUID();
+321 -40
View File
@@ -3,7 +3,7 @@ import path from "node:path";
import { execFile as execFileCallback } from "node:child_process";
import { promisify } from "node:util";
import { randomUUID } from "node:crypto";
import { and, asc, desc, eq, getTableColumns, gt, inArray, isNull, lt, lte, notInArray, or, sql } from "drizzle-orm";
import { and, asc, desc, eq, getTableColumns, gt, gte, inArray, isNull, lt, lte, notInArray, or, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
AGENT_DEFAULT_MAX_CONCURRENT_RUNS,
@@ -32,6 +32,7 @@ import {
approvals,
companySkills as companySkillsTable,
companies,
costEvents,
documentAnnotationComments,
documentAnnotationThreads,
documentRevisions,
@@ -238,6 +239,7 @@ const EXECUTION_PATH_HEARTBEAT_RUN_STATUSES = ["queued", "running", "scheduled_r
const CANCELLABLE_HEARTBEAT_RUN_STATUSES = ["queued", "running", "scheduled_retry"] as const;
const HEARTBEAT_RUN_TERMINAL_STATUSES = ["succeeded", "failed", "cancelled", "timed_out"] as const;
const UNSUCCESSFUL_HEARTBEAT_RUN_TERMINAL_STATUSES = ["failed", "cancelled", "timed_out"] as const;
const TIMER_ACTIONABLE_ISSUE_STATUSES = ["todo", "in_progress"] as const;
export {
ACTIVE_RUN_OUTPUT_CONTINUE_REARM_MS,
ACTIVE_RUN_OUTPUT_CRITICAL_THRESHOLD_MS,
@@ -6835,9 +6837,169 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
intervalSec: Math.max(0, asNumber(heartbeat.intervalSec, 0)),
wakeOnDemand: asBoolean(heartbeat.wakeOnDemand ?? heartbeat.wakeOnAssignment ?? heartbeat.wakeOnOnDemand ?? heartbeat.wakeOnAutomation, true),
maxConcurrentRuns: normalizeMaxConcurrentRuns(heartbeat.maxConcurrentRuns),
skipTimerWhenNoActionableWork: asBoolean(
heartbeat.skipTimerWhenNoActionableWork ??
heartbeat.requireActionableTimerWork ??
heartbeat.issueOnlyTimer,
false,
),
maxDailyRuns: normalizeOptionalNonNegativeInteger(
heartbeat.maxDailyRuns ?? heartbeat.dailyRunLimit ?? heartbeat.dailyRunCap ?? heartbeat.maxRunsPerDay,
),
maxDailyCostCents: normalizeOptionalNonNegativeInteger(
heartbeat.maxDailyCostCents ??
heartbeat.dailyCostCentsLimit ??
heartbeat.dailySpendCentsLimit ??
heartbeat.dailyBudgetCents,
),
};
}
function normalizeOptionalNonNegativeInteger(value: unknown) {
if (value === null || value === undefined || value === "") return null;
const normalized = Math.floor(asNumber(value, 0));
return normalized >= 0 ? normalized : null;
}
function currentUtcDayWindow(now = new Date()) {
const start = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate(), 0, 0, 0, 0));
const end = new Date(Date.UTC(now.getUTCFullYear(), now.getUTCMonth(), now.getUTCDate() + 1, 0, 0, 0, 0));
return { start, end };
}
async function getHeartbeatDailyCapBlock(
agent: typeof agents.$inferSelect,
policy: ReturnType<typeof parseHeartbeatPolicy>,
options: { checkRunCap?: boolean; checkCostCap?: boolean; excludeRunId?: string | null } = {},
client: Pick<Db, "select"> = db,
) {
const checkRunCap = options.checkRunCap ?? true;
const checkCostCap = options.checkCostCap ?? true;
const { start, end } = currentUtcDayWindow();
if (checkRunCap && policy.maxDailyRuns !== null) {
const conditions = [
eq(heartbeatRuns.companyId, agent.companyId),
eq(heartbeatRuns.agentId, agent.id),
gte(heartbeatRuns.startedAt, start),
lt(heartbeatRuns.startedAt, end),
notInArray(heartbeatRuns.status, ["queued", "scheduled_retry"]),
];
if (options.excludeRunId) {
conditions.push(sql`${heartbeatRuns.id} <> ${options.excludeRunId}`);
}
const [row] = await client
.select({ total: sql<number>`count(*)::integer` })
.from(heartbeatRuns)
.where(and(...conditions));
const observed = Number(row?.total ?? 0);
if (observed >= policy.maxDailyRuns) {
return {
reason: "heartbeat.daily_run_limit",
observed,
limit: policy.maxDailyRuns,
};
}
}
if (checkCostCap && policy.maxDailyCostCents !== null) {
const [row] = await client
.select({ total: sql<number>`coalesce(sum(${costEvents.costCents})::bigint, 0)` })
.from(costEvents)
.where(
and(
eq(costEvents.companyId, agent.companyId),
eq(costEvents.agentId, agent.id),
gte(costEvents.occurredAt, start),
lt(costEvents.occurredAt, end),
),
);
const observed = Number(row?.total ?? 0);
if (observed >= policy.maxDailyCostCents) {
return {
reason: "heartbeat.daily_cost_limit",
observed,
limit: policy.maxDailyCostCents,
};
}
}
return null;
}
async function cancelQueuedRunForHeartbeatDailyCap(
run: typeof heartbeatRuns.$inferSelect,
dailyCapBlock: NonNullable<Awaited<ReturnType<typeof getHeartbeatDailyCapBlock>>>,
) {
const now = new Date();
const reason = "Cancelled because the agent reached a per-day heartbeat budget cap before adapter invocation";
const cancelled = await setRunStatus(run.id, "cancelled", {
finishedAt: now,
error: reason,
errorCode: dailyCapBlock.reason,
resultJson: {
...parseObject(run.resultJson),
stopReason: dailyCapBlock.reason,
observed: dailyCapBlock.observed,
limit: dailyCapBlock.limit,
effectiveTimeoutSec: 0,
timeoutConfigured: false,
timeoutSource: "heartbeat_daily_cap_gate",
timeoutFired: false,
},
});
if (!cancelled) return null;
await setWakeupStatus(run.wakeupRequestId, "skipped", {
finishedAt: now,
error: reason,
});
await appendRunEvent(cancelled, await nextRunEventSeq(cancelled.id), {
eventType: "lifecycle",
stream: "system",
level: "warn",
message: reason,
payload: {
reason: dailyCapBlock.reason,
observed: dailyCapBlock.observed,
limit: dailyCapBlock.limit,
},
});
await releaseIssueExecutionAndPromote(cancelled, { suppressImmediateRecovery: true });
return cancelled;
}
async function hasActionableTimerWork(agent: typeof agents.$inferSelect) {
const row = await db
.select({ id: issues.id })
.from(issues)
.where(
and(
eq(issues.companyId, agent.companyId),
eq(issues.assigneeAgentId, agent.id),
isNull(issues.assigneeUserId),
isNull(issues.hiddenAt),
inArray(issues.status, [...TIMER_ACTIONABLE_ISSUE_STATUSES]),
),
)
.limit(1)
.then((rows) => rows[0] ?? null);
return Boolean(row);
}
async function markTimerHeartbeatChecked(agentId: string, source: WakeupOptions["source"]) {
if (source !== "timer") return;
await db
.update(agents)
.set({
lastHeartbeatAt: new Date(),
updatedAt: new Date(),
})
.where(eq(agents.id, agentId));
}
function parseMaxTurnContinuationPolicy(agent: typeof agents.$inferSelect): MaxTurnContinuationPolicy {
const runtimeConfig = parseObject(agent.runtimeConfig);
const heartbeat = parseObject(runtimeConfig.heartbeat);
@@ -6915,6 +7077,16 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
return null;
}
const dailyCapBlock = await getHeartbeatDailyCapBlock(agent, parseHeartbeatPolicy(agent), {
excludeRunId: run.id,
checkRunCap: true,
checkCostCap: true,
});
if (dailyCapBlock) {
await cancelQueuedRunForHeartbeatDailyCap(run, dailyCapBlock);
return null;
}
const issueId = readNonEmptyString(context.issueId);
if (issueId) {
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(run.companyId, issueId);
@@ -9801,7 +9973,10 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
);
}
async function releaseIssueExecutionAndPromote(run: typeof heartbeatRuns.$inferSelect) {
async function releaseIssueExecutionAndPromote(
run: typeof heartbeatRuns.$inferSelect,
options: { suppressImmediateRecovery?: boolean } = {},
) {
const runContext = parseObject(run.contextSnapshot);
const contextIssueId = readNonEmptyString(runContext.issueId);
const taskKey = deriveTaskKeyWithHeartbeatFallback(runContext, null);
@@ -10188,6 +10363,9 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
if (!issueNeedsImmediateRecovery) {
return { kind: "released" as const };
}
if (options.suppressImmediateRecovery) {
return { kind: "released" as const };
}
const existingExecutionPath = await tx
.select({ id: heartbeatRuns.id })
@@ -10410,6 +10588,14 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
...patch,
});
};
const writeSkippedHeartbeatRequest = async (skipReason: string, details: Record<string, unknown>) => {
await writeSkippedRequest(skipReason, {
payload: {
...(payload ?? {}),
heartbeatSkip: details,
},
});
};
const company = await db
.select({ status: companies.status })
@@ -10523,6 +10709,20 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
return null;
}
const genericTimerWake =
source === "timer" &&
!issueId &&
!wakeCommentId &&
!readNonEmptyString(enrichedContextSnapshot.taskId) &&
!readNonEmptyString(enrichedContextSnapshot.taskKey);
if (policy.skipTimerWhenNoActionableWork && genericTimerWake && !(await hasActionableTimerWork(agent))) {
await writeSkippedHeartbeatRequest("heartbeat.timer.no_actionable_work", {
reason: "No assigned todo or in_progress issue requires this agent before timer adapter invocation.",
});
await markTimerHeartbeatChecked(agentId, source);
return null;
}
if (issueId) {
const activePauseHold = await treeControlSvc.getActivePauseHoldGate(agent.companyId, issueId);
if (activePauseHold) {
@@ -10996,6 +11196,41 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
}
}
const dailyCapBlock = await getHeartbeatDailyCapBlock(agent, policy, {}, tx);
if (dailyCapBlock) {
const now = new Date();
await tx.insert(agentWakeupRequests).values({
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: dailyCapBlock.reason,
payload: {
...(payload ?? {}),
heartbeatSkip: {
reason: "Per-agent heartbeat daily cap reached before adapter invocation.",
observed: dailyCapBlock.observed,
limit: dailyCapBlock.limit,
},
},
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: now,
});
if (source === "timer") {
await tx
.update(agents)
.set({
lastHeartbeatAt: now,
updatedAt: now,
})
.where(eq(agents.id, agentId));
}
return { kind: "skipped" as const };
}
const wakeupRequest = await tx
.insert(agentWakeupRequests)
.values({
@@ -11130,46 +11365,92 @@ export function heartbeatService(db: Db, options: HeartbeatServiceOptions = {})
return mergedRun;
}
const wakeupRequest = await db
.insert(agentWakeupRequests)
.values({
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason,
payload,
status: "queued",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
})
.returning()
.then((rows) => rows[0]);
const queueOutcome = await db.transaction(async (tx) => {
await tx.execute(
sql`select id from agents where id = ${agentId} and company_id = ${agent.companyId} for update`,
);
const newRun = await db
.insert(heartbeatRuns)
.values({
companyId: agent.companyId,
agentId,
invocationSource: source,
triggerDetail,
status: "queued",
wakeupRequestId: wakeupRequest.id,
contextSnapshot: enrichedContextSnapshot,
sessionIdBefore: sessionBefore,
continuationAttempt,
})
.returning()
.then((rows) => rows[0]);
const dailyCapBlock = await getHeartbeatDailyCapBlock(agent, policy, {}, tx);
if (dailyCapBlock) {
const now = new Date();
await tx.insert(agentWakeupRequests).values({
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason: dailyCapBlock.reason,
payload: {
...(payload ?? {}),
heartbeatSkip: {
reason: "Per-agent heartbeat daily cap reached before adapter invocation.",
observed: dailyCapBlock.observed,
limit: dailyCapBlock.limit,
},
},
status: "skipped",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
finishedAt: now,
});
if (source === "timer") {
await tx
.update(agents)
.set({
lastHeartbeatAt: now,
updatedAt: now,
})
.where(eq(agents.id, agentId));
}
return { kind: "skipped" as const };
}
await db
.update(agentWakeupRequests)
.set({
runId: newRun.id,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
const wakeupRequest = await tx
.insert(agentWakeupRequests)
.values({
companyId: agent.companyId,
agentId,
source,
triggerDetail,
reason,
payload,
status: "queued",
requestedByActorType: opts.requestedByActorType ?? null,
requestedByActorId: opts.requestedByActorId ?? null,
idempotencyKey: opts.idempotencyKey ?? null,
})
.returning()
.then((rows) => rows[0]);
const newRun = await tx
.insert(heartbeatRuns)
.values({
companyId: agent.companyId,
agentId,
invocationSource: source,
triggerDetail,
status: "queued",
wakeupRequestId: wakeupRequest.id,
contextSnapshot: enrichedContextSnapshot,
sessionIdBefore: sessionBefore,
continuationAttempt,
})
.returning()
.then((rows) => rows[0]);
await tx
.update(agentWakeupRequests)
.set({
runId: newRun.id,
updatedAt: new Date(),
})
.where(eq(agentWakeupRequests.id, wakeupRequest.id));
return { kind: "queued" as const, run: newRun };
});
if (queueOutcome.kind === "skipped") return null;
const newRun = queueOutcome.run;
publishLiveEvent({
companyId: newRun.companyId,