fix(heartbeat): prevent zombie run coalescing and ensure startup reap completes before timer ticks (#1731)

## Thinking Path

> - Paperclip orchestrates AI agents for zero-human companies
> - Agents run in heartbeats — short execution windows triggered by the
heartbeat service
> - The heartbeat service coalesces overlapping wakeups: if a run for an
agent is already active, a new wakeup merges into it rather than
creating a duplicate
> - But when the server restarts, in-progress runs are left in
`"running"` status in the database — their child processes are gone, but
the DB rows persist as orphans
> - The startup `reapOrphanedRuns()` was fired as a `void` promise — the
timer interval started immediately in parallel, so the first timer tick
could coalesce a new wakeup into an orphaned "running" row before the
reap had a chance to remove it
> - Once coalesced, the orphan's `updatedAt` refreshed, making the
reaper skip it as "not old enough" — a zombie run that prevents the
agent from ever waking again
> - This PR fixes both the coalescing guard (do not coalesce into a
zombie) and the startup ordering (await reap before starting the timer),
eliminating the death spiral

## What Changed

- **`server/src/index.ts`** — `startServer` now `await`s
`reapOrphanedRuns()` (with one retry) before calling `setInterval`.
Timer ticks cannot start until orphaned runs are cleaned up.
- **`server/src/services/heartbeat.ts`** — Added two exported pure
functions:
- `isZombieRun(run, tracked)` — returns `true` if a run is `"running"`
in the DB but has no live entry in the in-memory `runningProcesses` Map
- `filterZombieCoalesceTarget(target, tracked)` — returns `null` if the
coalesce candidate is a zombie, letting the wakeup fall through to
create a new queued run instead
- Both coalescing call sites now use `filterZombieCoalesceTarget` before
deciding to coalesce
- **`server/src/__tests__/heartbeat-zombie-guard.test.ts`** — 8 new
behavioral tests covering `isZombieRun` and
`filterZombieCoalesceTarget`, including the critical zombie scenario,
legitimate live runs, queued runs (must never be filtered), and null
pass-through

## Verification

```bash
# Run the new tests
pnpm test:run
```

Manual reproduction (before fix):
1. Start an agent on a timer heartbeat
2. Kill the server mid-run (child process dies, DB row stays
`"running"`)
3. Restart the server
4. Observe: agent never wakes again — subsequent wakeups coalesce into
the dead run, refreshing `updatedAt`, keeping it alive forever

After fix: startup reap clears the orphan before the timer starts;
subsequent wakeups create fresh queued runs.

The one pre-existing test failure (`worktree helpers > copies shared git
hooks`) is unrelated — it fails on `upstream/master` as well due to a
`pnpm install` failure in the test environment.

## Risks

- **Startup latency**: `await reapOrphanedRuns()` adds a small delay
before the timer starts. In practice this is a fast DB query. The retry
adds at most one extra attempt on transient failure.
- **Behavior change**: Wakeups that previously coalesced into zombie
runs will now create new queued runs instead. This is the correct
behavior — the zombie was preventing any forward progress.
- **Queued runs unaffected**: `isZombieRun` only flags `"running"`
status. Queued runs pass through `filterZombieCoalesceTarget` unchanged
(covered by tests).

## Model Used

- OpenAI GPT-5 via a Codex-style terminal coding agent with tool use and
git/gh access. Exact hosted alias is not exposed in this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge

## Cross-references and status (maintainer)

Refs #3168
Refs #4174
Refs #4697
Refs #6399

Related PRs checked: #4075, #4705, #5232, #6952
- [x] I have searched GitHub for duplicate or related PRs and linked
them above

---------

Co-authored-by: Devin Foley <devin@paperclip.ing>

server/src/index.ts

@@ -745,56 +745,73 @@ export async function startServer(): Promise<StartedServer> {
   if (config.heartbeatSchedulerEnabled) {
     const heartbeat = heartbeatService(db as any, { pluginWorkerManager });
     const routines = routineService(db as any, { pluginWorkerManager });
-  
-    // Reap orphaned running runs at startup while in-memory execution state is empty,
-    // then resume any persisted queued runs that were waiting on the previous process.
-    void heartbeat
-      .reapOrphanedRuns()
-      .then(() => heartbeat.promoteDueScheduledRetries())
-      .then(async (promotion) => {
-        await heartbeat.resumeQueuedRuns();
-        const reconciled = await heartbeat.reconcileStrandedAssignedIssues();
-        if (
-          promotion.promoted > 0 ||
-          reconciled.assignmentDispatched > 0 ||
-          reconciled.dispatchRequeued > 0 ||
-          reconciled.continuationRequeued > 0 ||
-          reconciled.successfulRunHandoffEscalated > 0 ||
-          reconciled.escalated > 0
-        ) {
-          logger.warn(
-            { promotedScheduledRetries: promotion.promoted, promotedScheduledRetryRunIds: promotion.runIds, ...reconciled },
-            "startup heartbeat recovery changed assigned issue state",
+
+    // Reap orphaned runs before timer ticks start so wakeups cannot coalesce
+    // into a dead "running" row during startup recovery.
+    await (async () => {
+      for (let attempt = 1; attempt <= 2; attempt++) {
+        try {
+          const result = await heartbeat.reapOrphanedRuns();
+          logger.info(
+            { reaped: result.reaped, runIds: result.runIds },
+            "startup reap of orphaned heartbeat runs complete",
           );
+          break;
+        } catch (err) {
+          if (attempt < 2) {
+            logger.warn({ err, attempt }, "startup reap failed, retrying");
+          } else {
+            logger.error(
+              { err },
+              "startup reap of orphaned heartbeat runs failed after retry — periodic reaper will serve as degraded backstop",
+            );
+          }
         }
-      })
-      .then(async () => {
-        const reconciled = await heartbeat.reconcileIssueGraphLiveness();
-        if (reconciled.escalationsCreated > 0) {
-          logger.warn({ ...reconciled }, "startup issue-graph liveness reconciliation created escalations");
-        }
-      })
-      .then(async () => {
-        const scanned = await heartbeat.scanSilentActiveRuns();
-        if (scanned.created > 0 || scanned.escalated > 0) {
-          logger.warn({ ...scanned }, "startup active-run output watchdog created review work");
-        }
-      })
-      .then(async () => {
-        const swept = await heartbeat.sweepStaleIssueLocks();
-        if (swept.cleared > 0) {
-          logger.warn({ ...swept }, "startup stale-lock sweeper cleared issue locks");
-        }
-      })
-      .then(async () => {
-        const reviewed = await heartbeat.reconcileProductivityReviews();
-        if (reviewed.created > 0 || reviewed.updated > 0 || reviewed.failed > 0) {
-          logger.warn({ ...reviewed }, "startup productivity reconciliation created or updated review work");
-        }
-      })
-      .catch((err) => {
-        logger.error({ err }, "startup heartbeat recovery failed");
-      });
+      }
+
+      const promotion = await heartbeat.promoteDueScheduledRetries();
+      await heartbeat.resumeQueuedRuns();
+      const reconciled = await heartbeat.reconcileStrandedAssignedIssues();
+      if (
+        promotion.promoted > 0 ||
+        reconciled.assignmentDispatched > 0 ||
+        reconciled.dispatchRequeued > 0 ||
+        reconciled.continuationRequeued > 0 ||
+        reconciled.successfulRunHandoffEscalated > 0 ||
+        reconciled.escalated > 0
+      ) {
+        logger.warn(
+          { promotedScheduledRetries: promotion.promoted, promotedScheduledRetryRunIds: promotion.runIds, ...reconciled },
+          "startup heartbeat recovery changed assigned issue state",
+        );
+      }
+
+      const issueGraphReconciled = await heartbeat.reconcileIssueGraphLiveness();
+      if (issueGraphReconciled.escalationsCreated > 0) {
+        logger.warn(
+          { ...issueGraphReconciled },
+          "startup issue-graph liveness reconciliation created escalations",
+        );
+      }
+
+      const scanned = await heartbeat.scanSilentActiveRuns();
+      if (scanned.created > 0 || scanned.escalated > 0) {
+        logger.warn({ ...scanned }, "startup active-run output watchdog created review work");
+      }
+
+      const swept = await heartbeat.sweepStaleIssueLocks();
+      if (swept.cleared > 0) {
+        logger.warn({ ...swept }, "startup stale-lock sweeper cleared issue locks");
+      }
+
+      const reviewed = await heartbeat.reconcileProductivityReviews();
+      if (reviewed.created > 0 || reviewed.updated > 0 || reviewed.failed > 0) {
+        logger.warn({ ...reviewed }, "startup productivity reconciliation created or updated review work");
+      }
+    })().catch((err) => {
+      logger.error({ err }, "startup heartbeat recovery failed");
+    });
+
     setInterval(() => {
       void heartbeat
         .tickTimers(new Date())