+ WjeRt^RIt^RIt^RIt^RIt^RIHt^RIH t ]P!R4t Rt RRlt RR ltR R ltR R ltRRltR RRlltRRltRRltRRltRRltRtRtRtRRltRtR#)!uPost-restart fleet verification for ``hermes update`` (``update_cmd_fleet`` sibling). After the restart phase: compare every live gateway's stamped code against the checkout, reconcile the pre-update plan against what the restart did, keep the restart owed (a ``gateway_restart`` follow-up, never a failed update — contract C3) or clear the obligation, and finalize the receipt. Restart-phase helpers stay in ``update_cmd_fleet`` and are read through it so their patches apply. N)suppress) _best_effortzhermes_cli.update_cmdg^@cV^8dQhRR/#returnN)formats"VC:\Users\johnn\AppData\Local\hermes\hermes-agent\hermes_cli\update_cmd_fleet_verify.py __annotate__r s77T7c^RIHpHpHpV!4'dV!4'gR#\ 4\ R4V!4F$wr4pV'dRMRp\ RV RV R24K& \ 4\ R 4\ R 4\ R 4\ 4\ R 4\ 4\ R 4R#)zlLegacy hermes.service fights hermes-gateway.service over the bot token; warn on every update until migrated.)has_legacy_hermes_units_find_legacy_hermes_unitssupports_systemd_servicesNu+⚠ Legacy Hermes gateway unit(s) detected:systemuserz z (z scope)z; These pre-rename units (hermes.service) fight the currentz< hermes-gateway.service for the bot token and cause SIGTERMz flap loops. Remove them with:z! hermes gateway migrate-legacyz) (add `sudo` if any are in system scope))hermes_cli.gatewayrrrprint)rrrnamepathis_sysscopes r _print_legacy_units_warningrssr % ' ',C,E,E G 7879F" TF#eWG,-: G GH HI +, G -. G 56r c0V^8dQhR\R\/#)r rows_expectedr)boollist)r s"r r r /sDTr c^RIHp\VRR4;'gRpV'gV!VPVR7#VPp\P !4\ ,p\P!R4V!WCR7pVUu.uFp\Wt4'gKVNK ppV'dEV'g=\;QJdRV4F 'gK RM RM !RV44'gV#\P !4V8g\\VR R 44'gKVFpRVR &K V#uupi) uFleet version rows, polled over a bounded settle window when runtimes are expected. Gateways need time to rewrite gateway_state.json; Windows resumes DETACHED (~10s boot), so a single 2s sleep reported "no rows" on healthy resumes. A "down" row may be a detached replacement still booting: poll until none remain or the deadline passes. Pre-restart PIDs make a gateway stopped WITHOUT verified replacement a DOWN row (owed restart) instead of no row at all. An ``unknown`` row whose pid is NOT a pre-restart pid is a successor that has not published its code identity yet (a relaunched gateway can sit ~10s between process start and its first runtime-status write, #112634) — keep polling; at the deadline it is flagged ``identity_pending`` so the matrix does not call it a pre-stamping gateway. )collect_fleet_versionsself_restart_pending_pidsN)pre_restart_pidsself_restart_pendingTg@c3H"TFqPR4R8HxK R#5i)statedownNget).0rows& r *_collect_fleet_snapshot..Fs1aX`QT'''2Bf2LX`s "Frestarted_scoped_unitsidentity_pendingr) hermes_cli.update_receiptrgetattrpre_restart_gateway_pids_time monotonic#_FLEET_PROBE_SETTLE_TIMEOUT_SECONDSsleep_fleet_row_identity_pendingany_restarted_units_gone) restartrrpendingpre_pids_fleet_deadlinesnapshotr) unstampeds && r _collect_fleet_snapshotr>/sAg:DAIITG %$==T[] ]//Hoo'*MMO  C)8b$,[HS0KC0ZSSH [ Icc1aX`1accc1aX`1a.a.aO ??  /3H!92>4@4@ *.&'!O\s D>"D>c0V^8dQhR\R\/#)rr)r)dictr)r s"r r r OsZZTZZr cVPR4R8wgVPR4'dR#VfR#VPR4VUu0uF&p\V\4'gK\V4kK( up9#uupi)uAn ``unknown`` row with no sha from a pid that did not exist at update start: a relaunched gateway still booting, not a gateway that predates version stamping. A surviving pre-restart pid (or no pid snapshot at all) is settled as-is — waiting cannot change what it publishes.r$unknowncode_shaFpid)r' isinstanceint)r)r!ps&& r r5r5Osh www9$ (;(; 775>2B!Y2BQjQRTWFX&#a&2B!Y YY!Ys A;'A;c$V^8dQhR\/#rrr)r s"r r r Zs4r c^RIHpV'gR#\VP4pVFpVP R4wrEpVP W$,RRRV.,^R7P p\RTP444pTPR 4R 8wdR#TPR 4R 9gKR# R # \\\P3dR#i;i)uTrue when every restarted systemd unit is LOADED in its scope and neither active nor activating: the successor died, nothing will publish a state stamp, so the settle poll should fail closed now instead of at the deadline. Anything inconclusive keeps waiting: no units, systemctl missing/slow, or ``LoadState=not-found`` — a unit name asked in a scope that does not own it answers ``inactive`` exactly like a dead unit (#112466), so only a loaded unit can prove death.update_cmd_fleetF/showz-pzLoadState,ActiveState)timeoutc3X"TF pRV9gK VPR^4xK" R#5i)=N)split)r(lines& r r*(_restarted_units_gone..js*W4GD3RV;'TZZQ''4Gs ** LoadStateloaded ActiveStateT)active activating reloading) hermes_clirMr@_SYSTEMD_SCOPES partition _systemctlstdoutKeyErrorFileNotFoundError subprocessTimeoutExpired splitlinesr') scoped_unitsfleet scope_cmdsscopedr_rr`propss & r r7r7Zs 5 e++,J))#.$ %%j&764I`bf:g&gqr%szzFWF4E4E4GWW 99[ !X - 99] #'L L +Z-F-FG  s.C C+*C+c0V^8dQhR\R\/#)r fleet_rowsr)rr@)r s"r r r rsdtr c6/pVFpVPR4VPR4rCV'gK.\V\4'gKFVPR4R8wgK^VP\ V4\ 44P V4K V#)u3Profile -> gateway PIDs alive after the restart phase, from the fleet snapshot. Incarnation evidence for gateway reconciliation (``match_runtime_outcomes``'s ``live_gateway_pids``): the plan identifies a gateway by the profile it SERVES while the restart bookkeeping names the SERVICE, and a service can serve a profile its name does not encode — the profile-scoped name matcher then never credits the planned runtime. A ``down`` row reports the PRE-restart PID (nothing replaced it), so it is never a successor; rows without a usable profile/PID are skipped. rDprofiler$r%)r'rErF setdefaultstrsetadd)rmliver)rDros& r _live_gateway_pids_from_fleetrursrDwwu~swwy'9W 7z#s++0@F0J OOCL#% 0 4 4S 9 Kr cV^8dQhRR/#rr)r s"r r r s&^&^d&^r c ^RIHpVP4pV'dVP4M/pVfR#.pVP R4R9d9\ P !VR,4pVPV4TR,p^RI H p\VRR4;'gRUu.uFwp\VRR4R8XgK\\VR R4\4'gK:VPP4'gK\VPR 8wgKoV!V4NKy up\!V4U u.uF p RRR RR V R R /NK up ,p V U u.uF qV9gK V NK p p V 'dPV'dVP RR4MVP#4;'gRp VP%WV ,R7R#R# \dR#i;iuupiuup iuup i)a1Name the pre-update gateways on the obligation once their restart is owed. The pull arms the obligation before the restart phase knows what it owes. An inventory-less record on a host whose gateway then died at boot is settled by the gateway-less discharge (nothing live, nothing named), which would silence the owed-restart warning that replaces a failed update under contract C3. A named inventory keeps it until those gateways run HEAD. A gateway stopped with no known profile is recorded as an unmapped row for the same reason: it has no ``gateway_state.json`` the gateway-less probe could see, so absence would settle it. A readable inventory (a same-SHA catch-up keeps the standing record) only GROWS: rows this run newly owes are appended, so a mapped successor cannot discharge a newly stopped unmapped one. rLN inventoryruntimesasdictkindgatewayrorBrDstopped_unmappedT expected_sha)rrzNrynullr)r\rM_fleet_restart_obligation_armed_obligation_fieldsr'jsonloads_marker_owed_gateways ValueError dataclassesr|r/rErqrostripsorted_current_checkout_sha#_write_fleet_restart_pending_marker) planstopped_unmapped_pidsrgarmedfieldsrecordedrxr|runtimerDrowsr)shas && r _record_owed_gateway_inventoryrs5  1 1 3E+0U % % 'bF ~H zz+&88  6+#67I  ' ' 2Z("'.tZ'D'J'J'J 'JG 7FD )Y 6  ww 48# >  OO ! ! # )09(D w'J  01 31S )YeS:Ld S1 3  3D 74Ch#6CC4D 7 16fjj,ESY?Z;[7[[r cFV^8dQhR\\\3,/#rI)tuplerrr)r s"r r r sNNE$)$4Nr ca^RIHoVUu.uFqPR4R8XgKVNK ppVUu0uFJqPR4S9gK\VPR4\4'gKAVR,kKL pp\ ;QJd#V3RlV4F'gK RV3# RV3#!V3RlV44V3#uupiuupi) zY``(gateway missed, serve/dashboard PIDs missed)`` among the unaccounted runtime outcomes.) _SERVE_KINDSoutcome unaccountedr}rDc3J<"TFqPR4S9xK R#5i)r}Nr&)r(ors& r r*%_split_unaccounted..sA&QuuV}L0&s #TF)hermes_cli.update_inventoryrr'rErFr6)outcomesrmissed serve_pidsrs& @r _split_unaccountedrs8! GAUU9%5%FaaF G$*nFqeeFm|.K(PZ[\[`[`af[gilPm(!E((FJn 3A&A33: MM3: MM3A&A A: MMHnsCCC "C 7 C zserve/dashboard zJgateways may still run pre-update code or were stopped without a successorz'; recover with `hermes gateway restart`cHV^8dQhR\R\R\R\/#)r gateway_owed stale_serves down_servesr)rrrrq)r s"r r r s1JJDJPSJbeJjmJr c X.pV'dSTP\V'd$RRP\\V44 R2MR,\ ,4\ VPUu0uF0p\V4'gK\ V4PR4kK2 up4pV'd+VP\ RRPV4 R24V'dBVP\ RRP\\\ V444 R 24W4, 'dHVP\ RRP\\\ W4, 444 R 24R PV4;'g\\ ,#uupi) a`The ``gateway_restart`` follow-up's reason, naming what is actually owed (#141021). The step id stays ``gateway_restart`` (Desktop hand-offs parse it); only the text changes. A run whose gateways all verified current but whose dashboard unit could not be restarted must not tell the user to run `hermes gateway restart`, which never touches that unit.z (stopped PIDs z, )ryrzunit(s) zE were not restarted onto the new code; see the systemctl advice abovez process(es) zb were stopped during the update and not brought back; relaunch `hermes serve` / `hermes dashboard`z9 still run pre-update code; restart them as advised abovez; ) append_GATEWAY_OWED_REASONjoinrrq_GATEWAY_RECOVERYrrrrSERVE_ONLY_REASON_PREFIX)r8stoppedrrrpartsuunitss&&$$$ r _owed_restart_reasonrsg E DK3sG+a@QS U  ! W=Z=Zp=Z^lmn^o3CF'' 3=Zp qE  '(51A0BC. . / '( TYYs3{H[?\5]4^_Y Y Z!! '( TYYs3|OiHj?k5l4mn> > ? 99U  I I 47H HIqs 0F'F'c ^RIHp^RIHpHpHp^RIHp\R4;_uu_4\4RRR4\V4p V!\VP4\VP4,R7p V 'dRVn\4p Rp \R4;_uu_4V!V4p V 'd V!V 4RRR4\4\R 4\R 4.p \R 4;_uu_4^R IHpVP%4wppV!4P'WW PV4p\)VV4p V!V 4'd(R;Vnp ^R IHpV!WVP/44M$V 'gV'd\R4R;Vnp RRR4\R4;_uu_4VEeVP0'Ed^RIHpHp^RIHp\;V 4pT!TVPVP<VP>VP@VPV e"V Uu0uFpVPCR4kK upMRT V Uu0uF$pV!V4'gKVPCR4kK& upVR7 p^RI"H#p^RI$H%p\MVP0V4F3wppVR,R8XgKV!V!V4RR7'gK.RVR&K5 V!V4'd(RVn\OV4wppV V,p V V,p \Q\R4;_uu_4^RIH*pVPVPC4p V eVV PXR&RRR4RRR4VP'gA\R4;_uu_4\[4'd\R4R;Vnp RRR4\]VRR4'd R;Vnp VP'dx^RIH/p!\a\]VRR4;'gR+4p"T!!R\cTT"Y\T ;'gR+4R 74\R!4;_uu_4\eVV"4RRR4\R"4;_uu_4^R#IH3p#V#!R$V R%7p$V$e\hPkR&V$4RRR4VP'dR#VPm4V'gR#^R'I7H8p%V%!4R# +'giEL;i +'giELD;i +'giELz;iuupiuupi +'giEL;i +'giEL;i +'giEL;i +'giEL;i +'giL;i \R\r3d1p&\hPuR(T&4\R)T& R*24Rp&?&R#Rp&?&ii;i),aPost-restart verification: legacy-unit warning, dashboard cleanup, stale serve probe, fleet version matrix, plan-vs-execution reconciliation, receipt finalize. Never fails the committed update (contract C3): when any gateway may still be stale it records a ``gateway_restart`` follow-up and leaves ``fleet_restart_pending`` armed for the next catch-up and the startup warning; otherwise clears the marker. rL)_m$_surviving_pre_update_serve_runtimes_warn_stale_serve_runtimes)_refresh_dashboard_after_updatez*Legacy unit check during update failed: %sN)already_restarted_unitsTz0Failed to check for surviving serve runtimes: %sz-Tip: You can now select a provider and model:z7 hermes model # Select provider and modelz%Fleet version verification failed: %s)print_fleet_version_matrix)signal_stale_fleet_survivorsuq ⚠ Fleet version check returned no rows even though gateway runtimes were expected — verification incomplete.z)Runtime-outcome reconciliation failed: %s)match_runtime_outcomesreport_unaccounted_runtimes)row_is_externalrD) restarted_servicesrelaunched_profilesexternally_supervised_profiles killed_pids failed_unitsstale_serve_pidsfailed_respawn_pidsexternal_gateway_pidslive_gateway_pidsr{)defer_manual_serverr) require_alivedeferredruntime_outcomeszOwed-gateway check failed: %suW ⚠ A gateway owed a restart by a previous update is still not serving this checkout.r)record_followupgateway_restart)rrrz(Fleet restart inventory not recorded: %sz"Update receipt finalize failed: %s)finalize_update_receiptsuccess)rgzUpdate receipt written: %s)maybe_auto_migrate_after_updatez0Multiplex auto-migration after update failed: %su2 ⚠ Gateway multiplex migration did not finish: z% (run `hermes gateway migrate` later)r);r\rMhermes_cli.update_cmdrrrhermes_cli.update_cmd_maintrrrrrrrrrrr.rfleet_probe_signals_fleet_probe_expected_runtimesr>%hermes_cli.update_cmd_stale_survivorsr_gateway_drain_budgetrzrrrrrurrrr'rr|#hermes_cli.update_serve_obligationsrziprr Exceptionupdate_receipt_currentdatarr/rrrrrloggerinfo#_clear_fleet_restart_pending_markerhermes_cli.gateway_migrater SystemExitwarning)'r8_pre_update_plan_windows_gateway_resumeupdate_completergrrrr _gateway_owed_dashboards_down_stale_serves_owed_stale_serve_rows_fleet_snapshotr _pre_restart_killed_fleet_rows_expectedrrrr_live_gateway_pidsr)_runtime_outcomesr|rrr_missed_gateway_missed_serves_ur_activerrr _receipt_pathrexcs'&$$$ r _verify_fleet_after_updater s5L B C C#% D08M 7 #G$>$> ?#gFcFcBd df!!e&* H I I@AQR  &'8 9 J  G 9: CD O = > >H!( ; ; = g!tBB ,CE_E_ah 2';OP %o 6 615 5G  [ (5C^C^C` a %9 O 26 5G C ?J A B B  ',<,E,E,E i A "?!O  6 #*#=#=$+$?$?/6/U/U#//$:: )40AA/@SWWU^/@A$4AP&i#TcdgTh~swwu~&i"4#! & + N$'(8(A(ACT$U 9%6;MfU\omq;r;r)3GI&%V++<==%)"2DEV2W/0 "n4")$$7,,**,&7HGLL!34 %W Cb     9 : :*++op599"];w/66.21] >*A4HNNBO)+? W=,223,5 6D E E *+;W EF : ; ;E0 Q  $ KK4m D < --/  oN')Y D C C: J I I ? > >xB 'j%$$W C B Bd; : :,F E E ; ;( z "oI3O B3%Glmnnos S,,TBTTT>UAU2T( UT-*T-?U-4T2 !U%U U.;+V V, S= T  T% ( U2 U = U U  U+ . U?  V W&%WW)r)__doc__rloggingrctimer1 contextlibrhermes_cli.update_cmd_commonr getLoggerrr3rr>r5r7rurrrrrrrrrr rr r rs 5   2 3 ',#7*@Z0$&^R 5X\N.c=J:|or