j5RdZddlZddlmZddlmZddlZejZejZ iZ eddZ edZ dZ d Zd Zd Zd Zd ZdZdZdZdZdZdZdZddZdZejZejZdZdZdZ dZ!dS)zQModel-facing execution services, independent of MMGP's offload/compilation hooks.N)contextmanager) ContextVarmmgp_execution_checkpoint)defaultc#Kt|} dVt|dS#t|wxYw)zESupply a local cancellation check for long MMGP residency operations.N)_execution_checkpointsetreset) checkpointtokens 4C:\Users\johnn\OneDrive\Documents\AI\mmgp\runtime.pyexecution_scopers^ " % %j 1 1E+ ##E*****##E****s >AcTt}| |dSdSN)rget)callbacks r r r s0$((**H c~t|t||fdS)aTie a shared execution resource to a model's GPU residency. prepare(device) runs before its first owner loads, outside graph warmup; release() runs after its last owner unloads and its graphs are destroyed. Register before profiling, with the same callback pair for models sharing the resource. Neither callback may retain the model or allocate resources at registration time. N) _resources setdefaultr add)modulepreparereleases r register_resourcer s7&#%%((,,gw-?@@@@@rc |j|}|j|D]}t|t j}| }|||rrddlm}| 5|dtj dtj dddn #1swxYwYdS)N) allocatorrcuda)modelsruntime_resources_resource_usersrweakrefWeakSetrrnormal_allocationstorchdevicercurrent_device)managermodel_idmodelresourceusersfirstrs r _load_resourcesr0+s+ N8 $E-h7OO**8W_5F5FGG  %  O # # # # # #--// O O EL1J1J1L1LMMNNN O O O O O O O O O O O O O O O OOs=AC  C C c|D]k}|j|}|j|D]N}t|}||||s|dt|=OldS)Nr)r r!r"rdiscard)r* model_idsr+r,r-r.s r _unload_resourcesr47s . .x(1(; . .H#''11E} MM%  . #H- . . .rcfd|D}i_|D]r\}}tx}j|<|D]@}||t|<|t |dAs|D]\}}||t|<dS)Nc>i|]}|tj|fS)r#ref).0r+r*s r z_register..Es* P P PXhW--x8 P P Prr7)r!itemsr modules_ownersupdaterr)r*r ownersr+r, resourcesrs` r _registerrADs P P P P P P PF "G!<<>>99%:=%%? G-h7mmoo 9 9F$X.GFO   Z^^FB77 8 8 8 8 9"<<>>**%)**rcttD]\}\}}||urt|=dSr)listr=r;)r*r reference_s r _unregisterrFQsM"&w}}"7"7  A 9;;' ! !  rczt|}|dS|d}|dn ||dfS)Nrr)r=r)rentryr*s r _ownerrIWsC KK  E }teAhjjG?44%((;;rcbddlm}t||rtjdSt |tjdSt |d}|"t |d}|tjdn|jS)zReturn the execution device, even while managed weights are on CPU. An unmanaged module uses its first parameter/buffer's device, or CPU if empty. Registration is removed when its offload manager is released. r) ExpertBankcpuNr) expertsrK isinstancer'r(rInext parametersbuffers)rrKtensors r execution_devicerS_s $#####&*%%#|E""" f~~!|F### &##%%t , ,F ~fnn&&--"(.5<   fmCrc|D] }d|_ dS)a[Mark modules that MMGP computes on the CPU, their weights kept in RAM, never in VRAM: a token table only gathers the rows of a prompt, the same values as on the GPU, without moving the whole table (2 GB for an umT5 or a Gemma 3) at each load. Their tensor inputs go to the CPU and their output comes back to the execution device. Call it on the loaded modules, before the offload profile: a quantized table (a module whose tensors are not plain) keeps the GPU path of its qtype. A module tied to a marked one (a head sharing the table) must be marked too, with unused when it never runs.TN) _mm_on_cpur<rs r lookup_on_cpurWps( !! !!rc|D] }d|_ dS)a Mark modules whose parameters MMGP keeps in reserved RAM mapped in the GPU's address space, never in VRAM: their kernels read the rows they need in place, over the bus, inside CUDA graphs too and with indices computed on the GPU (a token table read for a few tokens at each step). The parameters stay the modules' own, as CUDA views of their RAM, outside the blocks, base and budgets; the RAM is charged to the reserved RAM while it exists. Parameters of other modules sharing their tensors follow them: never mark a table that a module reads whole at each call (a head tied to it), which would read it over the bus. Call it on the loaded modules, before the offload profile; without CUDA, or without reserved RAM left, the modules keep their usual placement.TN) _mm_mappedrVs r map_in_ramrZzs(!! !!rc$ddlm}||S)zdTrue for a parameter kept in mapped RAM (map_in_ram): a CUDA view of RAM, whose reads cross the bus.r) is_mapped)mappedr\)rRr\s r r\r\s$!!!!!! 9V  rc|D] }d|_ dS)aCMark modules that never run, such as the output head of a language model whose hidden states condition a generator (called through its base model): MMGP keeps their weights in RAM, out of VRAM, pinning and budgets, quantized or not, and raises if one runs. Call it on the loaded modules, before the offload profile.TN) _mm_unusedrVs r unusedr`s(!! !!rc`t|}||\}}||dSdS)zLoad a managed model's base/preloaded weights; leave block swapping to MMGP. Use before custom entry points that read weights without calling a hooked forward. Unmanaged modules are unchanged. This does not pin a model in VRAM. N)rIensure_model_loadedrownerr*r+s r ensure_loadedresA 6NNE !##H-----rstateFc >ddlm}t}|L|tt jdtjd|S|\}tfd|j  DdzfdttD}|tt jdtj|||| S) aA state ring (mmgp.state_ring.StateRing) for tensors a model computes once per block and reads at every pass (a cached attention prefix), blocks: the model's blocks (a ModuleList, called one by one). Kept in RAM and streamed block by block on the transfer stream of the model's offload object, with the weights, following MMGP's schedule for these blocks (its prefetch window; records locked for the blocks whose weights are pinned, then, for a model MMGP pins, for those kept in VRAM, as far as the reserved RAM allows; the others moved ahead through MMGP's staging ring with their blocks): each block's forward fetches its state. The model calls store(block, tensors) once per block, and release() at the end. resident: blocks whose weights MMGP keeps in VRAM may keep their state there too, in the room their preload leaves (see mmgp.state_ring; the model then reads the lists store() returns at each pass).r) StateRingNr)r<c3*K|] \}}|u |VdSrr7)r9namerblockss r zstate_ring..s3hh<4W]agWgWg$WgWgWgWghhr.c"g|] }d| S)/r7)r9indexr+prefixs r zstate_ring..s,MMM(,,V,U,,MMMr)r<r+entriesresident) state_ringrhrIlenr'r(rr)rOr named_modulesrange) rkrjrtrhrdr*rsr+rqs ` @@r rurus-&%%%%% 6NNE }yVel65:;T;T;V;V&W&WY]_cmsttttGX hhhh7>(+C+Q+Q+S+Shhh h hkn nFMMMMM%F :L:LMMMG 9S[[%,vuz7P7P7R7R"S"SU\^blr~FPW& ( ( ((rcBt}|dS|\}|jvrdSt|}| ix}t|<|j}|t f}|vdz|urdn/t fd|Ddzzfd|j Dx}|t f<|j tfd |DS) aWhether the weights of a managed module are all in VRAM now (its model loaded, none of the blocks under the module streamed; expert banks aside): an MMGP graph stage that runs this module may then span several of its blocks (see cuda_graphs.run). For a submodule (e.g. the language model of a larger model), only its own blocks count: blocks the stage never runs may stay out of VRAM. Check it at each call: a dynamic preload can release blocks between calls. Unmanaged modules: True.NTFror%c3*K|] \}}|u |VdSrr7)r9rjmrs r rlz#weights_resident..s4B|B|GDRSpqu{p{p{4p{p{p{p{B|B|rrmcxg|]6\}}|||tdzd7S)rN) startswithrv)r9rjsizer+rqs r rrz$weights_resident..s2j2j2jztUYIM2jRVRaRabhRiRi2j$s8}}q7H7I7I2J2j2j2jrc3 K|]}|vV dSrr7r9rj preloadeds r rlz#weights_resident..s(44Tty 444444r) rIactive_models_ids _block_namesrr idrOrwblocks_of_modules_sizesr;preloaded_blocks_per_modelall) rrdr*namesr,rkr+rqrs ` @@@r weights_residentrs~ 6NNE }tGXw000u   W % %E }(** W% N8 $E YY"V**- . .F ~C522dB|B|B|B|W\WjWjWlWlB|B|B|>|>|@C?CD2j2j2j2j2j]d]|^C^C^E^E2j2j2j j"V**-.28.s2__tY9X^K^K^dK^K^K^K^__rrmc4g|]}t|zSr7)str)r9rprqs r rrz!resident_runs..s$EEE#e**$EEErcXg|]&}jdz|zr|nd'SroN)rr)r9rjr*r+s r rrz!resident_runs..s~FfFfFfX\gNmNqNqrz~AsADHsHOIOIGSddOSFfFfFfrrFc g|] }|dup|v Srr7rs r rrz!resident_runs..s' A A A4TT\ .TY. A A Ar)rIrv_tower_entriesrrr rOrwrxrr auto_preloadplansr/appendtuple)rrkrdrsr,rflagsplanrunsstartrpr*r+rqrs ` @@@@r resident_runsrsW 6NNE }CKK&((GX  ((G'!*Gh(???x(____%2E2E2G2G_____beeEEEE%F 2D2DEEE-/[[(,CFfFfFfFfFf`eFfFfFf,g g.( AJEw000CKK'))28">eEl"> KKuU|4 5 5 5E ;;rct|}|dS|\}}|j|}|dup|S)aPlace a managed model's optional GPU expert copies now, outside graph recording (e.g. at the first step of a decoding loop replayed as one graph, which MMGP's hooks never see): as many as its budget or the dynamic preload allows, ordinary blocks first; never shrinks them. Returns False while its ordinary blocks are not all resident yet (call again at a later step). Models without experts: True.NT)rIexpert_residencyrplace)rrdr*r+ residencys r place_expertsrsS 6NNE }tGX(,,X66I   1  1 11rc`t|}|dS|\}}|jduo ||jjvS)zWhether an automatic VRAM preload manages this model's blocks: it keeps as many as fit and releases some when another allocation needs VRAM (a co-tenant loaded beside it then takes only what it needs), keeping them again later as they run.NF)rIrrrcs r dynamic_preloadrsA 6NNE }uGX  t + VG.s<rrJD$tx#~..74rrrrrrr)rIrrranyrr;rcs @r requires_layer_offloadrs 6NNE }uGX7+++t'H8L8R,R,Rt rrrr'JiJoJoJqJqrrr r rrr)rfF)"__doc__r# contextlibr contextvarsrr'WeakKeyDictionaryr=rr"rrr rr0r4rArFrIrSrWrZr\r`rerurrrrrrrr7rr rsWW%%%%%%""""""  $' # % % &W & ( ( " #>MMM+++ AAA O O O . . . * * *   <<<DDD"!!!!!! !!! . . .((((&555.)w(** **,,> 2 2 2WWWsssssr