## Thinking Path
> - Paperclip orchestrates AI agents for zero-human companies
> - Agents run in heartbeats — short execution windows triggered by the
heartbeat service
> - The heartbeat service coalesces overlapping wakeups: if a run for an
agent is already active, a new wakeup merges into it rather than
creating a duplicate
> - But when the server restarts, in-progress runs are left in
`"running"` status in the database — their child processes are gone, but
the DB rows persist as orphans
> - The startup `reapOrphanedRuns()` was fired as a `void` promise — the
timer interval started immediately in parallel, so the first timer tick
could coalesce a new wakeup into an orphaned "running" row before the
reap had a chance to remove it
> - Once coalesced, the orphan's `updatedAt` refreshed, making the
reaper skip it as "not old enough" — a zombie run that prevents the
agent from ever waking again
> - This PR fixes both the coalescing guard (do not coalesce into a
zombie) and the startup ordering (await reap before starting the timer),
eliminating the death spiral
## What Changed
- **`server/src/index.ts`** — `startServer` now `await`s
`reapOrphanedRuns()` (with one retry) before calling `setInterval`.
Timer ticks cannot start until orphaned runs are cleaned up.
- **`server/src/services/heartbeat.ts`** — Added two exported pure
functions:
- `isZombieRun(run, tracked)` — returns `true` if a run is `"running"`
in the DB but has no live entry in the in-memory `runningProcesses` Map
- `filterZombieCoalesceTarget(target, tracked)` — returns `null` if the
coalesce candidate is a zombie, letting the wakeup fall through to
create a new queued run instead
- Both coalescing call sites now use `filterZombieCoalesceTarget` before
deciding to coalesce
- **`server/src/__tests__/heartbeat-zombie-guard.test.ts`** — 8 new
behavioral tests covering `isZombieRun` and
`filterZombieCoalesceTarget`, including the critical zombie scenario,
legitimate live runs, queued runs (must never be filtered), and null
pass-through
## Verification
```bash
# Run the new tests
pnpm test:run
```
Manual reproduction (before fix):
1. Start an agent on a timer heartbeat
2. Kill the server mid-run (child process dies, DB row stays
`"running"`)
3. Restart the server
4. Observe: agent never wakes again — subsequent wakeups coalesce into
the dead run, refreshing `updatedAt`, keeping it alive forever
After fix: startup reap clears the orphan before the timer starts;
subsequent wakeups create fresh queued runs.
The one pre-existing test failure (`worktree helpers > copies shared git
hooks`) is unrelated — it fails on `upstream/master` as well due to a
`pnpm install` failure in the test environment.
## Risks
- **Startup latency**: `await reapOrphanedRuns()` adds a small delay
before the timer starts. In practice this is a fast DB query. The retry
adds at most one extra attempt on transient failure.
- **Behavior change**: Wakeups that previously coalesced into zombie
runs will now create new queued runs instead. This is the correct
behavior — the zombie was preventing any forward progress.
- **Queued runs unaffected**: `isZombieRun` only flags `"running"`
status. Queued runs pass through `filterZombieCoalesceTarget` unchanged
(covered by tests).
## Model Used
- OpenAI GPT-5 via a Codex-style terminal coding agent with tool use and
git/gh access. Exact hosted alias is not exposed in this environment.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] If this change affects the UI, I have included before/after
screenshots
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] I will address all Greptile and reviewer comments before
requesting merge
## Cross-references and status (maintainer)
Refs #3168
Refs #4174
Refs #4697
Refs #6399
Related PRs checked: #4075, #4705, #5232, #6952
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
---------
Co-authored-by: Devin Foley <devin@paperclip.ing>
Dale Carman
d782c4cd53