## Thinking Path
> - Paperclip orchestrates AI agents for zero-human companies
> - The `codex_local` adapter runs `codex exec` as a child process and
streams JSONL events from its stdout
> - NEE-79 caught a real-world `codex_local` orphan: codex sat in
`read()` on stdin for 1h+, no JSON events emitted past startup, no LLM
call in flight. The adapter had no inactivity timer; the only safety net
was the platform-level 1h silent-run detector
> - This is precisely the failure shape NEE-80 scoped: an
adapter-detected fault that should be killed *by the adapter* and
surfaced as `failed`, not waited out for an hour
> - This pull request adds an output-inactivity watchdog inside the
codex-local adapter that resets on every parsed JSONL event from stdout,
kills the child via SIGTERM → 5s grace → SIGKILL when it fires, and
resolves the run with a structured watchdog failure
> - The benefit is that NEE-79-class hangs shorter than 1h stop reaching
the platform-level safety net — they fail fast and visibly at the
adapter layer, with diagnostic logs that don't require host shell access
## Linked Issues or Issue Description
No existing GitHub issue covers this; describing the underlying bug
in-PR per the bug report template
(`.github/ISSUE_TEMPLATE/bug_report.yml`):
- **What happened?** A `codex_local` run sat in `read()` on stdin for
over an hour: codex emitted its startup JSONL events, then nothing — no
further events, no LLM call in flight, no exit. The adapter kept the
child alive indefinitely; the run was only reaped by the platform-level
1h silent-run safety net, an hour after it had effectively died.
- **Expected behavior:** The adapter should detect that its child has
stopped producing output long before the platform-level safety net, kill
it, and surface the run as `failed` with a diagnostic that explains what
happened.
- **Steps to reproduce:** Run any `codex_local` issue where `codex exec`
hangs after startup (e.g. codex blocks reading stdin and never emits
another JSONL event). Observe the run stays alive until the 1h platform
safety net fires.
- **Paperclip version or commit:** reproduced on master prior to this
branch.
- **Agent adapter(s) involved:** codex_local.
Related PRs found while searching for duplicates (none implement an
event-aware inactivity watchdog inside `codex_local`):
- #4004 — generic idle + wall watchdogs for `runChildProcess` in
adapter-utils; complementary, operates below the JSONL parse layer and
is not codex-event-aware
- #4742 — fail-fast on codex-local *startup* hang; this PR covers the
post-startup hang class
- #6861 — zombie-run termination for codex-local; reaping after exit,
not inactivity detection
- #7811 — the analogous output-idle timeout for grok-local
## What Changed
- `packages/adapters/codex-local/src/server/watchdog.ts` *(new)* — pure
watchdog primitive with injectable timers/clock:
`resolveCodexInactivityTimeout`, `createCodexInactivityWatchdog`,
`formatWatchdogErrorMessage`. Default `7 * 60_000` ms; honors `null` as
the disabled escape hatch
- `packages/adapters/codex-local/src/server/execute.ts` — `runAttempt`
now wraps `onSpawn` to capture `pid`/`processGroupId`, feeds stdout
chunks through `noteStdoutChunk`, and on watchdog fire sends SIGTERM to
the process group, schedules SIGKILL after 5s, and returns an
`AdapterExecutionResult` with `exitCode: null`, `signal:
SIGTERM|SIGKILL`, `errorMessage: "watchdog: no codex output for {N}m
{S}s"`, `errorCode: "codex_output_inactivity_watchdog"`. With
`errorMessage` set and `timedOut: false`, `heartbeat.ts:5860` maps the
run to `outcome === "failed"` (not `cancelled`)
- `packages/adapters/codex-local/src/index.ts` — `agentConfigurationDoc`
documents `outputInactivityTimeoutMs` (number ms; `null` disables;
non-positive falls back to default with a warning log at spawn)
- `packages/adapters/codex-local/src/server/watchdog.test.ts` *(new)* —
13 tests covering acceptance criteria 2 and 3 plus supporting cases:
fires after silence, no-fire across 12× (threshold − 1s) cycles,
multi-event chunks, non-JSON ignoring, single-fire idempotency,
formatter shape, full resolution table, `null` → disabled
-
`packages/adapters/codex-local/src/server/watchdog.integration.test.ts`
*(new)* — real Node subprocess that prints one JSONL event then sleeps;
`runChildProcess` reaps it within `threshold + 6s`; signal is SIGTERM or
SIGKILL; `parsedEventCount === 1` (acceptance criteria 1 and 4)
## Verification
```
$ pnpm --filter @paperclipai/adapter-codex-local typecheck
> tsc --noEmit
# clean
$ pnpm --filter @paperclipai/adapter-codex-local exec vitest run
✓ src/server/quota-spawn-error.test.ts (1 test)
✓ src/server/codex-home.test.ts (3 tests)
✓ src/server/codex-args.test.ts (3 tests)
✓ src/server/watchdog.test.ts (13 tests)
✓ src/server/parse.test.ts (9 tests)
✓ src/server/execute.remote.test.ts (4 tests)
✓ src/ui/parse-stdout.test.ts (3 tests)
✓ src/ui/build-config.test.ts (1 test)
✓ src/server/watchdog.integration.test.ts (1 test)
Test Files 9 passed (9)
Tests 38 passed (38)
```
Acceptance criteria check:
1. ✅ Simulated child emits one event then sleeps → killed at threshold;
result `errorMessage` matches `watchdog: no codex output for {N}m {S}s`
(`watchdog.integration.test.ts`)
2. ✅ Child emits events every (threshold − 1s) → not killed
(`watchdog.test.ts: "does not fire when events arrive every (threshold -
1s)"`)
3. ✅ `outputInactivityTimeoutMs: null` disables the watchdog
(`resolveCodexInactivityTimeout` returns `disabled`; `execute.ts` skips
watchdog construction and logs a startup warning)
4. ✅ Real subprocess reaped well within `threshold + 6s` — 250 ms
threshold, 290 ms wall clock in CI
5. ✅ Adapter-level fault → `outcome === "failed"` per
`heartbeat.ts:5860`. NEE-79-class hangs <1h get caught at the adapter,
not the platform-level safety net
Post-rebase verification (head `2a8703fab`, rebased onto master
`69a368ed5`):
```
$ pnpm --filter @paperclipai/adapter-codex-local typecheck # clean
$ pnpm --filter @paperclipai/adapter-codex-local exec vitest run
Test Files 11 passed (11)
Tests 64 passed (64)
```
## Risks
- Low risk. Behind a default-on watchdog that only fires after 7m of
zero parsed JSON events. Operators can disable it with
`outputInactivityTimeoutMs: null` for known-slow tasks
- The kill path reuses the same `process.kill(-pgid, signal)` pattern
that `runChildProcess` already uses for its terminal-result cleanup, so
signal semantics match the existing code path
- `timedOut: false` is preserved on watchdog fire — the platform-level
timeout outcome is unchanged, only the `failed`-vs-success
classification flips. No behavioral shift for already-failing runs
- Sandbox/SSH execution targets: the watchdog fires and emits the
structured log, but the kill is best-effort because remote pids aren't
owned by this process. The platform-level 1h safety net still applies.
Out of scope for NEE-81 by design
## Model Used
- Provider: Anthropic Claude
- Model: claude-opus-4-7
- Mode: Claude Code (extended thinking, tool use)
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] If this change affects the UI, I have included before/after
screenshots (N/A — no UI changes)
- [x] I have updated relevant documentation to reflect my changes
(`agentConfigurationDoc` for `outputInactivityTimeoutMs`)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (re-running on the rebased head)
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(P2 addressed; review threads resolved)
- [x] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Neeraj Kumar Singh <b.nirajkumarsingh@hotmail.com>
Co-authored-by: Devin Foley <devin@paperclip.ing>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Neeraj Kumar Singh B
7aa212296e