+ Wj!] 4t!Rt"R ]#R &R t$Rt%Rt&RVt']PP!4t)] PT!4PVt,RRlt-RRlt.] RRl4t/RRlt0RR/RRllt1RRlt2RRlt3RR lt4R!R"lt5R#R$lt6R%R&lt7R'R(lt8R)R/R*R+llt9R,R-lt:R.R/lt;R0R1ltR4RRR/R7lt?R8t@R9R:ltAR;R<ltBR=R>ltCR?tDR@tERARBltFRCRDltGRERFltHRGRHltIRIRRJ^2RKR/RLRMlltJRNROltKRPRQltLRRRSltMRTRUltNR#)WucProfile-local durable audit ledger for cron execution attempts. The ledger records what is known about each attempt; it is not a retry queue. Interrupted attempts become ``unknown`` only after their owner process is proved gone — a start-time reading that fails to match the claim-time fingerprint is not proof of death. Terminal states are immutable. ) annotationsN)contextmanager)datetime)Path)AnyDictIteratorListOptional)get_hermes_home)now)CLAIM_TTL_INACTIVITY_HEADROOM)record_cron_finishzOptional[Path]EXECUTIONS_FILEig>@g @cV^8dQhRR/#)returnsqlite3.Connection)formats"CC:\Users\johnn\AppData\Local\hermes\hermes-agent\cron\executions.py __annotate__r,s n n$ nc^RIHp^RIHp\;'g'\ 4P 4R, R, pV!VP4V!VRR\R7#))_ensure_cron_dir)open_dbcronz executions.dbzcron/executions.dbT)db_labelsynchronous_full initialize) cron.jobsrhermes_cli.sqlite_utilrrr resolveparent_initialize_schema)rrpaths r_connectr',sL+.  V V088:VCoUDT[[! 4"6Zl mmrc V^8dQhRRRR/#rconnrrNoner)rs"rrr8s**/*D*rc^RIHpVPR4V!VRRR4V!VRRR4VPR4VPR 4V!VRR R 4V!VRR R 4V!VRRR4VPR4R#)r)add_column_if_missingabCREATE TABLE IF NOT EXISTS executions ( id TEXT PRIMARY KEY, job_id TEXT NOT NULL, source TEXT NOT NULL, process_id TEXT NOT NULL, pid INTEGER NOT NULL, process_started_at INTEGER, status TEXT NOT NULL CHECK(status IN ('claimed','running','completed','failed','unknown')), handoff_pending INTEGER NOT NULL DEFAULT 0, handoff_started_at REAL, claimed_at TEXT NOT NULL, started_at TEXT, finished_at TEXT, error TEXT ) executionshandoff_pendingz*handoff_pending INTEGER NOT NULL DEFAULT 0handoff_started_atzhandoff_started_at REALzeCREATE INDEX IF NOT EXISTS idx_executions_job_claimed ON executions(job_id, claimed_at DESC, id DESC)zhCREATE INDEX IF NOT EXISTS idx_executions_status_claimed ON executions(status, claimed_at DESC, id DESC)delivery_outcomezdelivery_outcome TEXTscheduled_instantzscheduled_instant TEXT progress_atzprogress_at TEXTzvCREATE INDEX IF NOT EXISTS idx_executions_occurrence ON executions(job_id, scheduled_instant) WHERE status='completed'N)r"r-execute)r*r-s& rr%r%8s<LL $ l-4 l02K LL : LL :$ .@BYZ$ .AC[\$ m=OPLL LrcV^8dQhRR/#)rrzIterator[sqlite3.Connection]r)rs"rrrfs2rc #"^RIHp\;_uu_4V!\44;_uu_4pVxRRR4RRR4R# +'giL;i +'giR#;i5i)r) transactionN)r"r7_lockr')r7r*s r _transactionr9es62  HJ''4 (''s1A.AA A A. A A A+ % A.c$V^8dQhRRRRRR/#)rr*r execution_idstrrOptional[dict[str, Any]]r)rs"rrrms#22#232;S2rcfVPRV34P4pVe \V4#R#)#SELECT * FROM executions WHERE id=?N)r4fetchonedict)r*r;rows&& r_fetchrCms1 ,,<|o N W W YC491T1rr1c$V^8dQhRRRRRR/#)rrecordr=r1 Optional[str]rr+r)rs"rrrrs$   $  ;H     rcJ^RIHpV!WR7R# \dR#i;i)zFProject durable state to monitoring without affecting ledger behavior.)emit_execution_stater1N)agent.monitoring.cron_healthrH Exception)rEr1rHs&$ r_emit_execution_staterLrs$ EVG    s  ""c V^8dQhRRRR/#)rpidintr Optional[int]r)rs"rrr~sS]rcD^RIHpV!V4# \dR#i;i)r)get_process_start_timeN)gateway.statusrRrK)rNrRs& r_process_start_timerT~s'9%c** s  c$V^8dQhRRRRRR/#)rrNrO started_atrPrboolr)rs"rrrs!>>>>4>rc^RIHpV!V4'gR#TfT\P!48H#\ T4pTfR#^RIHpT!Y4# \dR#i;i)r) _pid_existsFT)start_time_fingerprints_match)rSrYrKosgetpidrTrZ)rNrVrYcurrentrZs&& r_owner_is_liver^sl.3 biik!!!#&G= ( == sA A#"A#cV^8dQhRR/#)rrzOptional[float]r)rs"rrrsrc^RIHp^RIHp\ V!44p\ P !V4'dV^8:dR#\V\,\ V!44\4#)uAge past which a claimed/running row with a LIVE owner is treated as wedged. Derived from the existing knobs, never a bare wall-clock constant: ``max(3 × HERMES_CRON_TIMEOUT, cron script timeout, 7200)``. Returns ``None`` (never reclaim live owners — today's behaviour) when the inactivity timeout is 0/unlimited or not a finite positive number: with no bound to derive from, fail closed. )_cron_inactivity_seconds)_get_script_timeoutN) cron.schedulerracron.scheduler_scriptrbfloatmathisfinitemaxr $LIVE_OWNER_STALE_CLAIM_FLOOR_SECONDS)rarb inactivitys r_live_owner_stale_after_secondsrksU89/12J == $ $ a 22 !#$, rc V^8dQhRRRR/#)r claimed_atr<rrer)rs"rrrsPP3P5Prch\4\P!V4, P4#)zZSeconds since ``claimed_at`` (NOT NULL, always the aware ISO string from hermes_time.now).) _hermes_nowr fromisoformat total_seconds)rms&r_claim_age_secondsrrs# MH22:> > M M OOrc$V^8dQhRRRRRR/#)rrmr<r3rFrrer)rs"rrrs!9939]9u9rc,\T;'gT4#)a/Seconds since the owner last proved it was making progress. A wedged worker (#115692) stops stamping ``progress_at``; a long but healthy run keeps stamping it, so the derived stale bound measures silence, not run length. Rows written before the column existed have no stamp and fall back to claim age. )rr)rmr3s&&r_stale_age_secondsrus k77Z 88rc V^8dQhRRRR/#)rr;r<rrWr)rs"rrrs ! !3 !4 !rc  \4P4p\4;_uu_4pVPRW\\ P !434pVP^8HuuRRR4# +'giR#;i)zStamp ``progress_at`` on a running attempt this process owns. Returns False when the row is no longer ours or no longer running (the caller treats that as informational only).z|UPDATE executions SET progress_at=? WHERE id=? AND status IN ('claimed','running') AND process_id=? AND pid=?N)ro isoformatr9r4 _PROCESS_IDr[r\rowcount)r;r r*curs& rtouch_execution_progressr|s[ - ! ! #C 4ll \  RYY[ 9  ||q s ;A11 B c V^8dQhRRRR/#r)r)rs"rrrs  ,  rc ZVPR\^\\4434R#)a8DELETE FROM executions WHERE id IN ( SELECT id FROM executions WHERE status IN ('completed','failed','unknown') ORDER BY julianday(finished_at) DESC, finished_at DESC, julianday(claimed_at) DESC, claimed_at DESC, id DESC LIMIT -1 OFFSET ? )N)r4rhrOMAX_TERMINAL_EXECUTIONS)r*s&r_prune_unlockedrs(LL  Q+, -/rr2c(V^8dQhRRRRRRRR/#)rjob_idr<sourcer2rFrzdict[str, Any]r)rs"rrrs, 4Arc^RIHp\4P4p\P !4P p\P!4p\4;_uu_4pVPRV\V4\V4\V\V4WC!V434\Wu4pRRR4\X4V# +'giL;i)zO,P R  + &! M s %A C C c$V^8dQhRRRRRR/#)rr;r<instantrFrr+r)rs"rrrs& U U3 U U4 Urc ^RIHp\4;_uu_4pVPRV!V4V\\ P !434pVP^8wd \R4hRRR4R# +'giR#;i)zGBind the store-claimed snapshot before a provider hands it to a worker.rzzUPDATE executions SET scheduled_instant=? WHERE id=? AND status='claimed' AND handoff_pending=0 AND process_id=? AND pid=?z2Cron occurrence could not be bound before dispatchN) rr2r9r4ryr[r\rz RuntimeError)r;rr2r*r{s&& rset_execution_occurrencersc2 4ll ? w '{BIIK P  <<1 ST T  s AA44 B c V^8dQhRRRR/#rr;r<rr=r)rs"rrrs9Qrc F\4;_uu_4pVPR\P!4V\\P !434pVP ^8wd RRR4R#\W4pRRR4\X4V# +'giL;i)zDFence restart recovery while an external worker is adopting a claim.zUPDATE executions SET handoff_pending=1, handoff_started_at=? WHERE id=? AND status='claimed' AND process_id=? AND pid=?N) r9r4timeryr[r\rzrCrL)r;r*r{rEs& rmark_execution_handoff_pendingrsx 4ll /YY[, RYY[ A   <<1  + &! M sAB0 B B c V^8dQhRRRR/#rr)rs"rrr s#2Jrc h\P!4p\V4p\4P 4p\ 4;_uu_4pVP R\WW034pVP^8wd RRR4R#\W@4pRRR4\X4V# +'giL;i)u Atomically transfer and start an attempt in its worker process. The dispatching gateway creates the row before spawning a restart-safe worker. Adoption is the single ``claimed`` → ``running`` gate: only the winner may acknowledge ownership or run side effects. aUPDATE executions SET process_id=?, pid=?, process_started_at=?, status='running', started_at=?, handoff_pending=0, handoff_started_at=NULL WHERE id=? AND status='claimed' AND handoff_pending=1N) r[r\rTrorxr9r4ryrzrCrL)r;rNprocess_started_atr r*r{rEs& radopt_claimed_executionr s ))+C,S1 - ! ! #C 4ll H #3 E   <<1  + &! M s ,B! B!! B1 c V^8dQhRRRR/#rr)rs"rrr%s1Irc N\4P4p\4;_uu_4pVPRW\\ P !434pVP^8wd RRR4R#\W 4pRRR4\X4V# +'giL;i)z7Transition one claimed attempt to running exactly once.zUPDATE executions SET status='running', started_at=?, handoff_pending=0, handoff_started_at=NULL WHERE id=? AND status='claimed' AND handoff_pending=0 AND process_id=? AND pid=?N) rorxr9r4ryr[r\rzrCrL)r;r r*r{rEs& rmark_execution_runningr%s - ! ! #C 4ll /  RYY[ 9   <<1  + &! M s?B4 B B$ errorc ,V^8dQhRRRRRRRRRR /#) rr;r<successrWrrFr1rr=r)rs"rrr8s6#'0=#rc \4P4pV'dRMRpV'dRMV'd \V4MRp\4;_uu_4p\\ P !43pVPRWTWcV.VO54p V P^8wd RRR4R#\V4\Wp4p RRR4\X VR7\W4V # +'giL*;i)zWrite a terminal result once; terminal attempts cannot be rewritten. Only the owning process may finish its attempt; receipt recovery for a departed owner settles through ``recover_receipted_execution``. completedfailedNunknown failureaUPDATE executions SET status=?, finished_at=?, error=?, handoff_pending=0, handoff_started_at=NULL, delivery_outcome=? WHERE id=? AND status IN ('claimed','running') AND process_id=? AND pid=?rI) rorxr<r9ryr[r\r4rzrrCrLr) r;rrr1r statusdetailr*ownerr{rEs &$$$ rfinish_executionr8s - ! ! #C#[FT3u: A z}Scheduler restarted after this execution's owner exited before a durable terminal state; whether side effects ran is unknown.zOwner process is still alive but the claim outlived the derived stale bound; treated as wedged (#115692). The process was not terminated; whether side effects ran is unknown.c(V^8dQhRRRRRRRR/#)rr;r<rrrr+r)rs"rrrs(>>S>#>c>d>rc\VRVR7R# \P\3d#p\P RYY#4Rp?R#Rp?ii;i)zClose the receipt of a run that never started: a ``claimed`` row never resolves. Best-effort so a ledger write cannot mask the failure the caller logs or re-raises.F)rrz7Job '%s': failed to close execution receipt %s (%s): %sN)rsqlite3ErrorOSErrorloggerr)r;rr record_errs&&& rsettle_unstarted_executionrsE>uEB MM7 #> N5 > >>sAA  AcV^8dQhRR/#)rrrOr)rs"rrrsIIIrc(\4P4p^p.pRpRp\4;_uu_4pVPR4P 4pVEFvpVR,\ 8XdK\ p\\VR,4VR,4'd@V'g \4pRpVe \VR,VR ,4V8:dK\pVR ,p VR ,'d5V e1\P!4\V 4, \8dKVPR WVR ,VR,VR,VR,VR ,VR ,VR,VR ,3 4p WP, pV P'gEKL\!WWR ,4p V fEKeVP#V 4EKy V'd \%V4RRR4VFp \'V 4K V# +'giL&;i)zMark abandoned attempts unknown without scheduling retries: rows whose owner is provably dead, plus rows whose live owner holds a claim older than the derived stale bound (the process is not killed).NFzSELECT id, status, process_id, pid, process_started_at, handoff_pending, handoff_started_at, claimed_at, progress_at FROM executions WHERE status IN ('claimed','running')rrNrTrmr3r0r/aqUPDATE executions SET status='unknown', finished_at=?, error=?, handoff_pending=0, handoff_started_at=NULL WHERE id=? AND status=? AND process_id=? AND pid=? AND handoff_pending=? AND handoff_started_at IS ? AND claimed_at=? AND progress_at IS ?idr)rorxr9r4fetchallry_OWNER_GONE_REASONr^rOrkru_OWNER_WEDGED_REASONrreHANDOFF_ADOPTION_GRACE_SECONDSrzrCappendrrL) r changed recovered stale_afterstale_after_resolvedr*rowsrBreasonr0r{rEs rrecover_interrupted_executionsrs - ! ! #CG&(I$(K  4|| 8 (* C< K/'Fc#e*os3G/HII,"A"CK+/(')#l*;S=OPT__-!$%9!: %&&&2IIK%(:";;01,,>c$iXL8I3u:&'-A)B\"C $68 C || #G|||$i0%$$V,ef  D !w xf% N} s,A0H$AH3B(H!H:H H H c$V^8dQhRRRRRR/#rr;r<rrrWr)rs"rrrs!44444rc :\4P4p\4;_uu_4pVPRV34P 4pVeVR,R 9d RRR4R#VR,\ 8Xd RRR4R#\ \VR,4VR,4'd RRR4R#VR,pVR ,'d>Ve:\P!4\V4, \8d RRR4R#VPR W!VR ,VR,VR,VR,34pVP^8wd RRR4R#\W04p\V4RRR4\X4R # +'giL;i)uiRecord one attempt as ``unknown`` with a cause this process actually observed. ``recover_interrupted_executions`` sweeps every attempt whose owner is provably dead, and knows nothing but that absence — so all it can write is ``_OWNER_GONE_REASON``, which asserts a scheduler restart. A waiter that held the worker's ``Popen`` knows more: the owner was that external worker, and it exited with a known status. Without this, a manual run whose worker dies is filed as "Scheduler restarted ..." (a restart that never happened) and, because the sweep leaves the row terminal, the waiter reports success and never records the run — the job's ``fire_claim`` then blocks the next manual fire for the whole lease (#128509). The attempt stays ``unknown``, not ``failed``: whether side effects ran is still unknown. Only the CAUSE becomes truthful. Returns False — leaving the caller to fall back to the generic sweep — when the row is absent, already terminal, owned by this process, inside the handoff adoption grace, or owned by a live process: a worker that is still running must never be terminalized out from under itself. zSELECT id, status, process_id, pid, process_started_at, handoff_pending, handoff_started_at FROM executions WHERE id=?NrFrrNrr0r/zUPDATE executions SET status='unknown', finished_at=?, error=?, handoff_pending=0, handoff_started_at=NULL WHERE id=? AND status=? AND process_id=? AND pid=?rTclaimedrunning)rorxr9r4r@ryr^rOrrerrzrCrrL)r;rr r*rBr0r{rEs&$ rterminalize_dead_ownerrsb& - ! ! #C 4ll -O  (* ;#h-/EE  |  +  #c%j/3/C+D E E !!56 ! " "". e$677:XX' (ll E#d)S]C 4Es5z R   <<1 9 :+= >&! A s05F *F *F ;F 3F AF F  F c$V^8dQhRRRRRR/#rr)rs"rrr-s!CCDrc \4P4p\4;_uu_4pVPRW!\ V4\ \ P!434pVP^8wd RRR4R#\V\ V44p\V4RRR4\X4R# +'giL;i)a]The firer itself gave up on its in-flight attempt (``CronExecutionUnknown``): record it ``unknown`` now, owner-fenced to this process, instead of leaving it ``running`` until a restart's dead-owner sweep. Receipt recovery never acts for a same-process running row, so the owner's later terminal receipt can settle it only once it is marked given up.zUPDATE executions SET status='unknown', finished_at=?, error=?, handoff_pending=0, handoff_started_at=NULL WHERE id=? AND status IN ('claimed','running') AND process_id=? AND pid=?NFT) rorxr9r4r<ryr[r\rzrCrrL)r;rr r*r{rEs&$ rgive_up_executionr-s - ! ! #C 4ll \#l+["))+ F H <<1  c,/0 &!  sA B3> B33 C rlimitbefore_claimed_atc(V^8dQhRRRRRRRR/#)rrrFrrOrrzlist[dict[str, Any]]r)rs"rrrAs,''',/'IV''rc n.p.pVe,VPR4VP\V44Ve4VPR4VP\V4.^,4V'dRRPV4,MRpVP\ ^\ \ V4R444\4;_uu_4pVPRV,R ,V4P4pRRR4XUu.uFp\V4NK up# +'giL+;iuupi) zFReturn indexed, newest-first execution history with cursor pagination.Nzjob_id=?z7(julianday(claimed_at), claimed_at) < (julianday(?), ?)z WHERE z AND izSELECT * FROM executionszF ORDER BY julianday(claimed_at) DESC, claimed_at DESC, id DESC LIMIT ?) rr<extendjoinrhminrOr9r4rrA) rrrclausesparamswherer*rrBs $$$ rlist_executionsrAsGF z" c&k"$PQ s,-.2318I W- -bE MM#aSZ-./ 4|| & .V W   (*   "& &#DI &&  's /DD2 D/ c V^8dQhRRRR/#rr)rs"rrr[s222(@2rc\4;_uu_4pVPR\V434P4pRRR4Xe \ V4#R# +'giL!;i)zBReturn one exact execution attempt, or ``None`` when it is absent.r?N)r9r4r<r@rA)r;r*rBs& r get_executionr[sU 4ll 1     (*   491T1 s +A A' c V^8dQhRRRR/#rrr<rr=r)rs"rrres%%S%%=%rcB\V^R7pV'd V^,#R#))rrN)r)rrs& rlatest_executionres & 2D47$$rc V^8dQhRRRR/#rr)rs"rrrjs  C ,D rc\V4pV'dVPR4R9dR#\\VR,4VPR44'gR#V#)ueThe job's latest attempt while it is still claimed/running under a LIVE owner, else ``None``. This is scheduler OWNERSHIP, not recent activity: a run inside a long tool call writes no heartbeat yet stays owned, while a run whose process died (watchdog kill, crash) does not. Read-only — unlike ``recover_interrupted_executions`` it never rewrites a row. rNrNrr)rgetr^rO)rrEs& rlive_inflight_executionrjsMf %F VZZ)1GG #fUm,fjj9M.N O O Mrc V^8dQhRRRR/#)rjob_idsz list[str]rzdict[str, dict[str, Any]]r)rs"rrrys66y6-F6rc\PV4Uu.uFq'gK \V4NK ppV'g/#RPRV44p\ 4;_uu_4pVP RV R2V4P 4pRRR4XUu/uFqfR,\V4bK up#uupi +'giL7;iuupi)z1Load latest execution for many jobs in one query.,c3&"TFpRxK R#5i)?Nr).0_s& r $latest_executions..~s/ACsagSELECT e.* FROM executions e WHERE e.id IN ( SELECT id FROM ( SELECT id, ROW_NUMBER() OVER ( PARTITION BY job_id ORDER BY julianday(claimed_at) DESC, claimed_at DESC, id DESC ) AS rn FROM executions WHERE job_id IN (z )) WHERE rn=1)Nr)rAfromkeysr<rr9r4r)rrclean placeholdersr*rrBs& rlatest_executionsrys'+}}W'= H'=V[S['=E H  88///L 4||6 7C^D  !    (*  15 5M49 $ 55% I  6s B:B:+%B?C? C )rrr)O__conditional_annotations____doc__ __future__rloggingrfr[r threadingrr contextlibrrpathlibrtypingrrrr r hermes_constantsr hermes_timer rocron.constantsr /hermes_cli.observability.shared_metrics_gatewayr getLogger__name__rr__annotations__rrri_TERMINAL_STATESRLockr8rrryr'r%r9rCrLrTr^rkrrrur|rrrrrrrrrrrrrrrrrrrrrrr)rs@rrs#  %66,*8N   8 $ #'&!%'-$5jjl n*Z2  KO  >$*P 9 ! DH. U"4&@D&*>_c.5F4 "; >IX4n('#'24'Y]'42% 6r