fix(heartbeat): prevent zombie run coalescing and ensure startup reap completes before timer ticks (#1731)

## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - Agents run in heartbeats — short execution windows triggered by the
heartbeat service
> - The heartbeat service coalesces overlapping wakeups: if a run for an
agent is already active, a new wakeup merges into it rather than
creating a duplicate
> - But when the server restarts, in-progress runs are left in
`"running"` status in the database — their child processes are gone, but
the DB rows persist as orphans
> - The startup `reapOrphanedRuns()` was fired as a `void` promise — the
timer interval started immediately in parallel, so the first timer tick
could coalesce a new wakeup into an orphaned "running" row before the
reap had a chance to remove it
> - Once coalesced, the orphan's `updatedAt` refreshed, making the
reaper skip it as "not old enough" — a zombie run that prevents the
agent from ever waking again
> - This PR fixes both the coalescing guard (do not coalesce into a
zombie) and the startup ordering (await reap before starting the timer),
eliminating the death spiral

## What Changed

- **`server/src/index.ts`** — `startServer` now `await`s
`reapOrphanedRuns()` (with one retry) before calling `setInterval`.
Timer ticks cannot start until orphaned runs are cleaned up.
- **`server/src/services/heartbeat.ts`** — Added two exported pure
functions:
- `isZombieRun(run, tracked)` — returns `true` if a run is `"running"`
in the DB but has no live entry in the in-memory `runningProcesses` Map
- `filterZombieCoalesceTarget(target, tracked)` — returns `null` if the
coalesce candidate is a zombie, letting the wakeup fall through to
create a new queued run instead
- Both coalescing call sites now use `filterZombieCoalesceTarget` before
deciding to coalesce
- **`server/src/__tests__/heartbeat-zombie-guard.test.ts`** — 8 new
behavioral tests covering `isZombieRun` and
`filterZombieCoalesceTarget`, including the critical zombie scenario,
legitimate live runs, queued runs (must never be filtered), and null
pass-through

## Verification

```bash
# Run the new tests
pnpm test:run
```

Manual reproduction (before fix):
1. Start an agent on a timer heartbeat
2. Kill the server mid-run (child process dies, DB row stays
`"running"`)
3. Restart the server
4. Observe: agent never wakes again — subsequent wakeups coalesce into
the dead run, refreshing `updatedAt`, keeping it alive forever

After fix: startup reap clears the orphan before the timer starts;
subsequent wakeups create fresh queued runs.

The one pre-existing test failure (`worktree helpers > copies shared git
hooks`) is unrelated — it fails on `upstream/master` as well due to a
`pnpm install` failure in the test environment.

## Risks

- **Startup latency**: `await reapOrphanedRuns()` adds a small delay
before the timer starts. In practice this is a fast DB query. The retry
adds at most one extra attempt on transient failure.
- **Behavior change**: Wakeups that previously coalesced into zombie
runs will now create new queued runs instead. This is the correct
behavior — the zombie was preventing any forward progress.
- **Queued runs unaffected**: `isZombieRun` only flags `"running"`
status. Queued runs pass through `filterZombieCoalesceTarget` unchanged
(covered by tests).

## Model Used

- OpenAI GPT-5 via a Codex-style terminal coding agent with tool use and
git/gh access. Exact hosted alias is not exposed in this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

## Cross-references and status (maintainer)

Refs #3168
Refs #4174
Refs #4697
Refs #6399

Related PRs checked: #4075, #4705, #5232, #6952
- [x] I have searched GitHub for duplicate or related PRs and linked
them above

---------

Co-authored-by: Devin Foley <devin@paperclip.ing>

server/src/__tests__/heartbeat-comment-wake-batching.test.ts

@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
 import { createServer } from "node:http";
 import { and, asc, eq } from "drizzle-orm";
 import { WebSocketServer } from "ws";
-import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
 import {
   agents,
   agentWakeupRequests,
@@ -12,6 +12,7 @@ import {
   issueComments,
   issues,
 } from "@paperclipai/db";
+import { runningProcesses } from "../adapters/index.js";
 import { heartbeatService } from "../services/heartbeat.ts";
 import { SUCCESSFUL_RUN_HANDOFF_REQUIRED_NOTICE_BODY } from "../services/recovery/index.ts";
 import { startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.ts";
@@ -161,6 +162,10 @@ describe("heartbeat comment wake batching", () => {
     await tempDb?.cleanup();
   });
 
+  afterEach(() => {
+    runningProcesses.clear();
+  });
+
   it("defers approval-approved wakes for a running issue so the assignee resumes after the run", async () => {
     const companyId = randomUUID();
     const agentId = randomUUID();
@@ -201,6 +206,11 @@ describe("heartbeat comment wake batching", () => {
         wakeReason: "issue_assigned",
       },
     });
+    runningProcesses.set(runId, {
+      child: {} as never,
+      graceSec: 0,
+      processGroupId: null,
+    });
 
     await db.insert(issues).values({
       id: issueId,

server/src/__tests__/heartbeat-zombie-guard.test.ts

@@ -0,0 +1,125 @@
+import { describe, expect, it } from "vitest";
+import {
+  isZombieRun,
+  filterZombieCoalesceTarget,
+} from "../services/heartbeat.ts";
+
+// ---------------------------------------------------------------------------
+// isZombieRun — the core predicate
+// ---------------------------------------------------------------------------
+describe("isZombieRun", () => {
+  it("returns true for a running run not tracked in runningProcesses", () => {
+    const run = { status: "running", id: "run-1" };
+    const tracked = new Map<string, unknown>();
+
+    expect(isZombieRun(run, tracked)).toBe(true);
+  });
+
+  it("returns false for a queued run not tracked in runningProcesses", () => {
+    const run = { status: "queued", id: "run-2" };
+    const tracked = new Map<string, unknown>();
+
+    expect(isZombieRun(run, tracked)).toBe(false);
+  });
+
+  it("returns false for a running run that IS tracked in runningProcesses", () => {
+    const run = { status: "running", id: "run-3" };
+    const tracked = new Map<string, unknown>([["run-3", { pid: 12345 }]]);
+
+    expect(isZombieRun(run, tracked)).toBe(false);
+  });
+
+  it("returns false for a failed run not tracked in runningProcesses", () => {
+    const run = { status: "failed", id: "run-4" };
+    const tracked = new Map<string, unknown>();
+
+    expect(isZombieRun(run, tracked)).toBe(false);
+  });
+
+  it("returns false for a completed run not tracked in runningProcesses", () => {
+    const run = { status: "completed", id: "run-5" };
+    const tracked = new Map<string, unknown>();
+
+    expect(isZombieRun(run, tracked)).toBe(false);
+  });
+});
+
+// ---------------------------------------------------------------------------
+// filterZombieCoalesceTarget — the coalescing guard used in both paths
+//
+// These tests exercise the BEHAVIOR described in spec AC2 and AC3:
+// "Coalescing does not refresh updatedAt on zombie runs"
+// When the target is a zombie, the filter returns null so the wakeup
+// falls through to create a new queued run instead of merging into the dead one.
+// ---------------------------------------------------------------------------
+describe("filterZombieCoalesceTarget", () => {
+  // Bug 1 scenario: a "running" run with no live process is a zombie.
+  // Coalescing into it would refresh updatedAt, making it immortal.
+  it("returns null for a zombie running run (the critical bug fix)", () => {
+    const zombieRun = { status: "running", id: "zombie-1" };
+    const emptyTracked = new Map<string, unknown>();
+
+    expect(filterZombieCoalesceTarget(zombieRun, emptyTracked)).toBeNull();
+  });
+
+  // Legitimate running process — coalescing should proceed normally.
+  it("passes through a legitimate running run that IS tracked", () => {
+    const liveRun = { status: "running", id: "live-1" };
+    const tracked = new Map<string, unknown>([["live-1", { pid: 99 }]]);
+
+    expect(filterZombieCoalesceTarget(liveRun, tracked)).toBe(liveRun);
+  });
+
+  // Queued runs don't have processes yet — they must always pass through.
+  // isZombieRun only flags "running" status, so queued runs are safe.
+  it("passes through a queued run not tracked (queued runs are not zombies)", () => {
+    const queuedRun = { status: "queued", id: "queued-1" };
+    const emptyTracked = new Map<string, unknown>();
+
+    expect(filterZombieCoalesceTarget(queuedRun, emptyTracked)).toBe(queuedRun);
+  });
+
+  // null target means no candidate to coalesce into — pass through.
+  it("passes through null target unchanged", () => {
+    const tracked = new Map<string, unknown>();
+
+    expect(filterZombieCoalesceTarget(null, tracked)).toBeNull();
+  });
+
+  // Terminal states should never appear as coalesce targets, but if they do,
+  // they should pass through (they're not zombies — they're done).
+  it("passes through a failed run (terminal state, not a zombie)", () => {
+    const failedRun = { status: "failed", id: "failed-1" };
+    const emptyTracked = new Map<string, unknown>();
+
+    expect(filterZombieCoalesceTarget(failedRun, emptyTracked)).toBe(failedRun);
+  });
+
+  it("passes through a completed run (terminal state, not a zombie)", () => {
+    const completedRun = { status: "completed", id: "done-1" };
+    const emptyTracked = new Map<string, unknown>();
+
+    expect(filterZombieCoalesceTarget(completedRun, emptyTracked)).toBe(completedRun);
+  });
+
+  // Regression guard: after server restart, runningProcesses is empty.
+  // Multiple zombie runs should all be filtered to null.
+  it("filters multiple zombie runs independently (post-restart scenario)", () => {
+    const emptyTracked = new Map<string, unknown>();
+    const zombie1 = { status: "running", id: "z1" };
+    const zombie2 = { status: "running", id: "z2" };
+
+    expect(filterZombieCoalesceTarget(zombie1, emptyTracked)).toBeNull();
+    expect(filterZombieCoalesceTarget(zombie2, emptyTracked)).toBeNull();
+  });
+
+  // Mixed scenario: one zombie, one live. Only the zombie is filtered.
+  it("correctly distinguishes zombie from live when multiple runs exist", () => {
+    const tracked = new Map<string, unknown>([["live-1", { pid: 42 }]]);
+    const zombie = { status: "running", id: "zombie-1" };
+    const live = { status: "running", id: "live-1" };
+
+    expect(filterZombieCoalesceTarget(zombie, tracked)).toBeNull();
+    expect(filterZombieCoalesceTarget(live, tracked)).toBe(live);
+  });
+});