+ WjRa0t$Rt^RIHt^RIt^RIt^RIt^RIt^RIt^RI t ^RI H t H t H t ^RI Ht^RIHtHtHt]P&!]4tRtRt^KtRtR tR2t]!0R3m4tR tR t^t R t!R t"R t#Rt$Rt%Rt&]PN!4t(Rs)R]*R&RRlt+R4RRllt,RRlt-RRlt.RRlt/RRlt0RR lt1R!R"lt2!R#R$4t3R4R%]/R&R'lllt4R(R)lt5R*R+lt6R5R,R-llt7R6R.R/llt8R0R1lt9R#)7uStartup-liveness watchdog — respawn a gateway that wedges before its loop runs (OOF-298). Every other liveness backstop (loop-liveness watchdog, shutdown watchdog, heartbeat file) assumes startup succeeded; none can fire if the process deadlocks before the event loop is alive (OOF-298: ~30h with every thread in ``futex_wait_queue``, zero logs, s6 saw a live PID). A daemon thread armed at process entry and disarmed once the loop is confirmed live dumps all-thread stacks (``faulthandler``), records the exit in the lifecycle ledger (NS-608) and ``os._exit``\ s with the service-restart code so s6/systemd respawn. Slow-but-alive startups are not killed. Order of authority: (1) phase-owned progress leases (:func:`report_startup_progress`) — authoritative, prove the *startup path itself* is alive, work for I/O-bound phases (schema migration, corruption repair in ``SessionDB.__init__``) with ~zero CPU; (2) process-wide CPU progress, capped at ``_MAX_CPU_EXTENSIONS`` since an unrelated thread burning CPU must not hide a parked startup thread forever. Known limitation: a *spinning* startup deadlock earns the capped extensions before firing; the observed class is parked-thread deadlocks. Idle-by-design waits call :func:`kick_startup_watchdog` (respawn-storm backoff, up to 300s); MCP discovery's 120s wait sits inside the 300s default. IMPORT-LIGHTNESS IS A CORRECTNESS PROPERTY: top-level module, stdlib only. Arming must precede importing ``gateway`` (hundreds of modules; an import-time deadlock is in scope), and at fire time the wedged main thread may hold the import lock — so the fire path does no imports on its own thread; the ledger write runs on a helper thread joined with a timeout. Config is env-only (``HERMES_STARTUP_WATCHDOG=0``, ``HERMES_STARTUP_WATCHDOG_TIMEOUT_S``) because config.yaml parsing is itself in scope. Everything is best-effort: a watchdog failure must never affect the startup it observes. ) annotationsN)datetimetimezoneUTC)Path)AnyDictOptionalgr@g>@HERMES_STARTUP_WATCHDOG!HERMES_STARTUP_WATCHDOG_TIMEOUT_Sg@g?g @g$@armeddisarmedfiringOptional[StartupWatchdogHandle]_handlecV^8dQhRR/#)returnr)formats"KC:\Users\johnn\AppData\Local\hermes\hermes-agent\hermes_startup_watchdog.py __annotate__rds%%d%c^RIHpV!4#)z@Use the stdlib-only process resolver before application startup.get_process_hermes_home)hermes_constantsrrs r_process_hermes_homerds8 " $$rc V^8dQhRRRR/#)rhomezOptional[Path]rrr)rs"rrrks***4*rcHVeTM \4pVP!\#)z;Return ``/logs/gateway-startup-watchdog.log``.)rjoinpath_DUMP_RELATIVE)rbases& rget_startup_watchdog_dump_pathr$ks!#4)=)?D ==. ))rcV^8dQhRR/#rrboolr)rs"rrrqsOO4Orc\PP\R4P 4P 4\ 9#)z:True when ``HERMES_STARTUP_WATCHDOG`` opts out explicitly.)osenvirongetENV_STARTUP_WATCHDOGstriplower_FALSEYrrrstartup_watchdog_disabledr1qs/ ::>>. 3 9 9 ; A A Cw NNrcV^8dQhRR/#)rrfloatr)rs"rrrvs&&%&rcB\PP\R4P 4pV'g\ #\ V4pT^8:d\ #\T\4# \d*\PR\T\ 4\ u#i;i)zEDeadline in seconds; env override, floor-clamped, default on garbage.r)z/Ignoring non-numeric %s=%r; using default %.0fs) r*r+r,ENV_STARTUP_WATCHDOG_TIMEOUT_Sr."DEFAULT_STARTUP_WATCHDOG_TIMEOUT_Sr3 ValueErrorloggerwarningmax_MIN_TIMEOUT_S)rawvalues r resolve_startup_watchdog_timeoutr>vs **..7 < B B DC 112c  z11 un %% 2 = *C1S 21 2s A**1BBc V^8dQhRRRR/#)r failure_msgstrrNoner)rs"rrrs11S1T1rc\4pVPPRRR7\VRRR7;_uu_4pV!V4RRR4R# +'giR#;i \d\ P TRR7R#i;i)zQOpen the dump file for append and hand it to *write*; failures only log at DEBUG.T)parentsexist_okazutf-8)encodingNexc_info)r$parentmkdiropen Exceptionr8debug)writer@pathfhs&& r _append_dumprRsk1-/ $6 $g . ." "I/ . . . 1 [4 01s.=A& AA& A# A&#A&&"B  B c V^8dQhRRRR/#)rrecordzdict[str, Any]rrBr)rs"rrrs~$rc(a\V3RlR4R#)zDAppend a one-line JSON metadata record beside the faulthandler dump.ch<VP\P!S\R7R,4#))default )rOjsondumpsrA)rQrTs&r$_write_dump_record..s288DJJvs;dBCrz,Failed to write startup watchdog dump recordN)rR)rTsfr_write_dump_recordr]sC6rc$V^8dQhRRRRRR/#)rlevelrAmsgrrBr)rs"rrrs!   #  rc\\\V4!V.VO5!R# \dR#i;i)zPLog at *level*; never raises (the wedged main thread may hold the logging lock).N)getattrr8rM)r_r`argss&&*r _log_quietlyrds- s*T*    s  ++c V^8dQhRRRR/#)r exit_codeintrrBr)rs"rrrs   C  D  rcL^RIHpV!VRR7R# \dR#i;i)zRecord the watchdog exit in the NS-608 lifecycle sentinel. Runs on a helper thread (see ``_fire``): the import can block on the interpreter import lock, and the fire path must reach ``os._exit`` regardless. ) mark_exitedstartup_liveness_watchdog)reasonN)gateway.lifecycle_ledgerrirM)rfris& r_mark_lifecycle_exitrms'  8I&AB    s  ##c]tRt^tRtRRltRRltR"RRlltR#RR llt] R R l4t R R lt R$RRllt ] RRl4tRRltRRlt] RRl4t] RRl4tRRltRRltRR ltR!tR#)%StartupWatchdogHandlez>+2(037  +/#//+rcV^8dQhRR/#rrrBr)rs"rrrrs # # #rc VP;_uu_4VP\8Xd \VnRRR4VPP 4R# +'giL,;i)uStartup reached a live event loop — stand down. Idempotent. Atomic with respect to firing: whichever of disarm/fire takes the state lock first wins, so a disarm landing before the fire sequence is never lost. N)r|ryrx _DISARMEDrsetrs&rdisarmStartupWatchdogHandle.disarmsE    {{f$'    " s !A A* c V^8dQhRRRR/#rextra_sr3rrBr)rs"rrrrs G GE GD Grc *\R\V44pVP;_uu_4\ P !4VP,V,VnRRR4R# \\3dRpLfi;i +'giR#;i)zPush the deadline out to ``now + timeout + extra_s``. For call sites about to block intentionally with ~zero CPU (the respawn-storm backoff sleep), otherwise indistinguishable from a parked deadlock. rtN) r:r3 TypeErrorr7r|rurvrqr})rrextras&& rkickStartupWatchdogHandle.kicksn  U7^,E   !^^->FDN :& E    sA(3B(A>=A> B c$V^8dQhRRRRRR/#r expected_sr3phaserArrBr)rs"rrrrs!###c#4#rc \V4pT^8:dR#\T\4pTP ;_uu_4\ TP\P!4T,4TnT'd\T4Tn T;P^, un RRR4R# \\3dR#i;i +'giR#;i)u~Claim a progress lease: this phase expects up to ``expected_s`` more seconds of work. The authoritative "still making progress" signal — owned by the startup path, so it works for I/O-bound phases and cannot be counterfeited by unrelated threads. Clamped to ``_MAX_LEASE_S`` per call so one buggy caller cannot silence the watchdog indefinitely; long phases renew. Never raises. N) r3rr7min _MAX_LEASE_Sr|r:rrurvrArr)rrrexpecteds&&& rleaseStartupWatchdogHandle.leases Z(H q= x.     #D$5$5t~~7G(7R SD $'J!    "    :&      s B+A'C+C?C C cV^8dQhRR/#r&r)rs"rrrrs(($(rc (VP\8H#N)ryrrs&rr StartupWatchdogHandle.disarmeds{{i''rcV^8dQhRR/#r&r)rs"rrrrsDD$Drc bVPRJ;'dVPP4#r)ris_alivers&rrStartupWatchdogHandle.is_alives&||4'CCDLL,A,A,CCrNc V^8dQhRRRR/#)rtimeoutOptional[float]rrBr)rs"rrrrs//O/t/rc ^VPeVPPVR7R#R#)Nr)rjoin)rrs&&rrStartupWatchdogHandle.joins& << # LL  g  . $rcV^8dQhRR/#)rrrr)rs"rrrrs/rc R\P!4# \dR#i;i)zBProcess-wide CPU time (user+system, all threads); None on failure.N)ru process_timerMrrr_process_cpu_seconds*StartupWatchdogHandle._process_cpu_secondss( $$& &  s  &&cV^8dQhRR/#rr)rs"rrrrs<#<#t<#rc \P!VPRRR7P4\ P !4VP, p\RRVPWPVP4\R\P!\4P!4RRR \"P$!4R VPR \'V^4R VPR VP(RVP*RVP/ 4\-RR4\.P0!RR7\P!\6VP3RRR7pVP4VP9\:R7VP<P?4VPAVP4R# \dELi;i \d\2P5RRR7Li;i \dL~i;i)unForensics, then exit — with the exit itself independently bounded. Everything producing forensics (logging, dump record, faulthandler, ledger) can block: the wedged main thread may hold the logging handler lock, the disk may be hung. None of that may stop the respawn, so the escort thread starts FIRST. ``os._exit`` is async-signal-safe and lock-free by design. Tz$gateway-startup-watchdog-exit-escorttargetdaemonnamecriticalzGateway startup did not reach a live event loop within %.0fs (elapsed %.0fs, %d extension(s)), holds no progress lease and shows no CPU progress; dumping all thread stacks and exiting with code %d so the service supervisor can restart it (OOF-298).tstagzstartup_watchdog.firedpidrq elapsed_s extensions lease_countlast_lease_phaserfc2\P!VRR7#)T)file all_threads) faulthandlerdump_traceback)rQs&rr[-StartupWatchdogHandle._fire..0s|22Mrz4Startup watchdog file-based faulthandler dump failed)rz)Startup watchdog faulthandler dump failedrHzgateway-startup-watchdog-ledger)rrcrrrN)!rzThread _exit_escortstartrMrurvrwrdrqrrfr]rnowr isoformatr*getpidroundrrrRrrr8rNrmr_LEDGER_JOIN_TIMEOUT_Srr_exit)relapsed ledger_threads& r_fireStartupWatchdogHandle._fires    ((>2CD6M    !   '=  >  4>>"a   B U LLDtL T U   s61G&G>AG> GG"G;:G;> H  H cV^8dQhRR/#rr)rs"rrrrFs # #d #rc VP\4VPP4'dR#VP VP 4R#)zHard-exit if the forensic fire path wedges (bounded-exit seam). Deliberately free of log handlers, filesystem access, module loads and any lock shared with application code: only a sleep, an Event check and the exit seam. N)_sleep_FIRE_EXIT_BOUND_Sris_setrrfrs&rr"StartupWatchdogHandle._exit_escortFs7 &' ?? ! ! # #  4>>"rc V^8dQhRRRR/#)rsecondsr3rrBr)rs"rrrrRs$rc 2\P!V4R#)z4Seam for tests; production is a bare ``time.sleep``.N)rusleep)rs&rrStartupWatchdogHandle._sleepQs 7rc V^8dQhRRRR/#)rcodergrrBr)rs"rrrrWsCDrc 2\P!V4R#)z2Seam for tests; production is a bare ``os._exit``.N)r*r)rs&rrStartupWatchdogHandle._exitVs rc V^8dQhRRRR/#)rdeadliner3rr'r)rs"rrrr[s4rc VP;_uu_4VP\8wd RRR4R#WnRRR4R# +'giR#;i)zDSet a new deadline under the state lock; False when no longer armed.NFT)r|ryrxr})rrs&&r_extend_if_armed&StartupWatchdogHandle._extend_if_armed[s>    {{f$ &N    sAA A cV^8dQhRR/#rr)rs"rrrrcs88d8rc  VP4pVP;_uu_4VP\8wd RRR4R#VPpRRR4X\ P !4, pV^8d5VPP\V\4R7'dR#K\ P !4pVP;_uu_4VPpVPpRRR4XV8d~VP\WT\\VP4,44'gR#\!RRVPX;'gRWT, 4VP4pEKoVP4pVe Ve Wq, MRpVeV\"8dVP$\&8dTpV;P$^, unVP\ P !4VP,4'gR#\!RRVPWP$\&4EK7VP;_uu_4VP\8wd RRR4R#\(VnRRR4VP+4R# +'giELX;i +'giEL;i +'giLJ;i)TNrr9ueGateway startup exceeded %.0fs but phase %r holds a progress lease for another %.0fs — honoring it.unknownuGateway startup exceeded %.0fs but is consuming CPU (%.1fs this window); extending the startup watchdog deadline (CPU-fallback extension %d of %d — phases doing long legitimate work should call report_startup_progress instead).)rr|ryrxr}rurvrwaitr _POLL_SLICE_Srrrr:rqrd_CPU_PROGRESS_MIN_Sr_MAX_CPU_EXTENSIONS_FIRINGr) rlast_cpur remainingr lease_until lease_phasecpu window_deltas & r_runStartupWatchdogHandle._runcs1,,.!!!;;&("! >>"!4>>#33I1}'',,SM5R,SS.."C!!!"// "// "S ,,SC W[WeWeDf>f-ghhHNNK$<$<9k>O  446++-C-0_AU3>[_L'LrorrMr8rN)rqrfresolvedhandles&$ rarm_startup_watchdogrs $ & & \"w'7'7'9'9\ (U9-=-Ai 57  +8?F==??\G\\\  = MsRC C "B8B8 C A B8 C *B8- C 8 C C C "C21C2cV^8dQhRR/#rr)rs"rrrsQQQrc\;_uu_4\pRsVeVP4RRR4R# +'giR#;i \d\P RRR7R#i;i)uDisarm the process-wide startup watchdog, if armed. Never raises. ``disarm()`` runs while still holding the singleton lock — it is non-blocking, and this closes the window where a concurrent re-arm could swap in a new handle that the disarm then misses. Nz)Failed to disarm gateway startup watchdogTrH)r rrrMr8rNrs rdisarm_startup_watchdogrsUQ \FG!  \\\ Q @4 PQs+A 7A A A A "A10A1c$V^8dQhRRRRRR/#)rmethodrAr@rrBr)rs"rrrs!11s111rc\;_uu_4\pRRR4Xe\W04!V!R#R# +'giL';i \d\P TRR7R#i;i)zVCall ``handle.(*args)`` on the armed handle; no-op when unarmed, never raises.NTrH)r rrbrMr8rN)rr@rcrs&&* r_with_armed_handlersV1 \F   F #T * \ 1 [4 01s%A 6A A A "A/.A/c V^8dQhRRRR/#rr)rs"rrrsSS5S4Src \RRV4R#)zExtend the armed watchdog's deadline. No-op when not armed; never raises. Call before intentionally blocking with ~zero CPU activity (e.g. the respawn-storm backoff sleep) so the idle wait is not mistaken for a parked deadlock. rz'Failed to kick gateway startup watchdogNr)rs&rkick_startup_watchdogrs vH'Rrc$V^8dQhRRRRRR/#rr)rs"rrrs& X X Xc X4 Xrc \RRW4R#)uDeclare a phase-owned progress lease on the armed startup watchdog. Call from startup phases about to do legitimately long synchronous work — most importantly ``state.db`` schema migrations and corruption repair/backup inside ``SessionDB.__init__`` — with an honest worst case, renewing for multi-step phases. Per-call duration is clamped to ``_MAX_LEASE_S``. No-op when not armed; never raises — safe to call unconditionally from application code. rz!Failed to report startup progressNr)rrs&&rreport_startup_progressrsw CZWrcV^8dQhRR/#rr)rs"rrrs$rc\;_uu_4\pRsRRR4XeVP4R#R# +'giL(;i)z0Drop the module singleton (test isolation only).N)r rrrs r_reset_for_testsr"s6    s 8 A )logszgateway-startup-watchdog.log>0noofffalserrr):__conditional_annotations__r __future__rrrYloggingr*rzrurrrpathlibrtypingrrr getLoggerrr8r6r;SERVICE_RESTART_EXIT_CODEr-r5r" frozensetr0rrrrrrrxrrr{r r__annotations__rr$r1r>rRr]rdrmrorrrrrr")r(s@rr1s/8#   ,,&&   8 $%*"0!D9 / 0         ~~ +/ (/%* O &$1   ttn/@Q$1S Xr