j)ddlmZddlmZddlmZmZddlZeGddZeGddZ Gd d Z dddZ GddZ GddZ dS)) annotations) dataclass)CallableIterableNceZdZdZd d Zd S) GraphKVIndexWriterzNWrites decode K/V values at the absolute token position provided by input_pos.k_cache torch.Tensorv_cachek_valv_val input_posreturnNonec|ddd}|jtjkr |tj}|j|jkr||j}|d|jdd|||d|jdd||dS)N)dtype)devicer)reshapertorchlongtorshape index_copy_)selfr r r r rindexs HC:\Users\johnn\OneDrive\Documents\AI\shared\llm_engines\cudagraph_kit.pywritezGraphKVIndexWriter.writes!!"%%bqb) ;%* $ $HH5:H..E <7> ) )HHGNH33E %+a. !--a>>> %+a. !--a>>>>>N) r r r r r r r r rr rr)__name__ __module__ __qualname____doc__rr rrr s.XX??????r rc"eZdZUdZdZded<dS)GraphKVFlashStatezGMarks a cache as using flash_attn_with_kvcache-compatible graph decode.Tbool flash_kvcacheN)r!r"r#r$r)__annotations__r%r rr'r's)QQMr r'c:eZdZdZddZdd Zdd ZddZddZdS)CUDAGraphRunnerz1Captures and replays a fixed-shape CUDA callable.namestrfnCallable[..., object]rrcL||_||_d|_g|_d|_dSN)_name_fn_graph_static_inputs_output)rr-r/s r__init__zCUDAGraphRunner.__init__"s* 37 24# r inputsr objectc*|std|jd|D]V}tj|st d|jd|jjdkrtd|jdWd|D|_tj |_ tj 5tj |j d5|j |j|_dddn #1swxYwYdddn #1swxYwYtj 5|j dddn #1swxYwYn##t $r|wxYw|jS) N[z5] CUDAGraphRunner requires at least one tensor input.z-] CUDAGraphRunner accepts tensor inputs only.cudaz(] CUDAGraphRunner requires CUDA tensors.cZg|](}|)Sr%)detachclone).0tensors r z,CUDAGraphRunner._capture..1s,LLL6v}}4466LLLr thread_local)capture_error_mode) ValueErrorr3r is_tensor TypeErrorrtyper6r= CUDAGraphr5inference_modegraphr4r7replay Exceptionrelease)rr9rBs r_capturezCUDAGraphRunner._capture)s dbbbbcc c [ [F?6** _ ]DJ ] ] ]^^^}!V++ !YTZ!Y!Y!YZZZ,LLVLLLj**,,  %'' B BZ%%dkn%UUBB#+48T-@#ADLBBBBBBBBBBBBBBB B B B B B B B B B B B B B B B %'' % % ""$$$ % % % % % % % % % % % % % % %    LLNNN  |sx)E+<'D#D8 DD D D D E+DE+"D#E+9E E+E##E+&E#'E++ F cNt|t|jkr=td|jdt|jdt|dt |j|D]\}}|j|jks |j|jks|j|jkrbtd|jdt|jd|jd|jdt|jd|jd|jd| |dS) Nr<z ] Expected z inputs, got .z"] Static shape mismatch: expected  z on z, got ) lenr6rFr3ziprrrtuplecopy_)rr9 static_tensorrBs r _copy_inputszCUDAGraphRunner._copy_inputs@s v;;#d122 2 2mmmD.gs4 R R Rt=uqy3Q3Q R RF R R R Rr ) rIrTensoris_cuda isinstancennModulelistrVdictvalueshasattrvars)valuer_itemss ` rrgrgYs E{{el""-/wwR/ qyyJueho66y %$'' E4    # #U ""$$ R R R Ru R R RRr ceZdZdZddZdS) _Stagea-A decoding step of the kit as an MMGP graph stage of its owner's model. MMGP keys a stage's graphs by the module (the owner, or the part of it that the steps run: the weights the step reads) and the method: one function per runner name, kept as an attribute of that module while the kit lives. attributer.rrcDd_|_dfd }||j|<dS)Nr9r rr:cj|Sr2)r/)r9rs rstepz_Stage.__init__..stepss47F# #r r])r/rwrd)rownerrwrzs` rr8z_Stage.__init__osA04" $ $ $ $ $ $$(y!!!r N)rwr.rr)r!r"r#r$r8r%r rrvrvjs2##))))))r rvc>eZdZdZdddZdd ZdddZddZddZdS) AutoRegressiveCudaGraphKitaReusable helper to attach graph-safe KV writers and replay decode callables. With an owner that MMGP manages with CUDA graphs (its handler lists it in cudaGraphs), a decoding step is an MMGP graph stage: one graph while the owner's weights are all in VRAM, otherwise the step runs with each layer replayed from its own MMGP graph, whether the layer is kept in VRAM or streamed. The caches the steps read go through register_state. Other owners keep the kit's own graphs, which need the weights in place. resident: the module the steps run when it is only a part of the owner (e.g. its language model): its weights alone key the graphs (the owner's other blocks may stream meanwhile) and its blocks in VRAM are enough for a single graph per step.Nr-r.rrc||_||_||n||_| d|j_i|_i|_d|_g|_t|_ dS)NTF) r3_owner _resident_mm_graph_placement_key_runners_stages_state_registered_attached_cachesset_attached_cache_ids)rr-r{residents rr8z#AutoRegressiveCudaGraphKit.__init__s`  "*"2  59DN 246 *, !&.0-0UU   r holdersr:c|jdSddlm}d|D}||j|j|d|Dd|_dS)ajThe CUDA tensors (KV caches, static buffers) that the steps read or update besides the owner's weights and the step inputs, at fixed addresses until release(): MMGP owners only (a no-op otherwise). Registering new tensors drops the graphs. The cache objects among the holders (e.g. a transformers StaticCache) may be passed to the owner's layers.Nr cuda_graphsc6g|]}t|D]}|Sr%rf)rAholderrBs rrCz=AutoRegressiveCudaGraphKit.register_state..s-TTTfmF>S>STTF6TTTTr c lg|]1}t|tjttt f/|2Sr%)rkrrirnrVro)rArs rrCz=AutoRegressiveCudaGraphKit.register_state..szNiNiNiY_r|~DGLGSUY[`bfFgshshNifNiNiNir )rT)rmmgprregister_stater3r)rrrtensorss rrz)AutoRegressiveCudaGraphKit.register_states ;  F$$$$$$TTTTT""4; GNiNicjNiNiNi" j j j!%r rattention_modulesIterable[object] max_tokensr` graph_modec|dkrtd|dvrtd|dd}|D]}t|dd}|td||t |t |j}t|}||jvra|dkrt|_ nt|_ |j ||j ||S) Nrzmax_tokens must be > 0.)rr)zUnsupported graph_mode 'z'.kv_cachez:KV cache must be initialized before CUDA graph attachment.r))rFgetattr RuntimeErrorensure_capacitymaxr`capacityidrr' graph_stateraddrappend)rrrrfinal_capacitymodulercache_ids rattach_attention_modulesz3AutoRegressiveCudaGraphKit.attach_attention_moduless  ??677 7 7 7 7F FFFGG G' 7 7Fvz488H"#_```  $ $Z 0 0 0 X5F1G1GHHN(||Ht77700+<+>+>H((+=+?+?H((,,X666%,,X666r runner_namer/r0r9r c2|jFddlm}m}|d5||jr|j|}|*t|j d|j d|x}|j|<||_ |j |j g|R|j ||j dcdddS dddn #1swxYwY||jrN|jD]}||j||S|j|}|$t)|j d||}||j|<|j |S) Nr)rruntimeF)gated_mm_kit__)methodenabled:)rrrrscope is_enabledrgetrvrr3r/r[rwweights_residentrequires_layer_offloadrrprOclearr,)rrr/r9rrstagerunners rr[zAutoRegressiveCudaGraphKit.runsZ ; " 1 1 1 1 1 1 1 1"""// N N))$+66N L,,[99E}Sx^b^hSxSxkvSxSxNFNN5?dkd|d|~B~LeMeMNNN  N N N N N N N NN N N N N N N N N N N N N N N N--dk:: #"m2244%%FNN$$$$ ##%%%r6{""";// >$ %B%B[%B%BBGGF)/DM+ &vz6""sBCC #C cZ|jD]}||j|jD]'}|jj|jd(|j|j r-ddl m }| |j |jd|_ |jD]}t!|drd|_|j|jdS)NrrFr)rrprOrrrrdpoprwrrr release_staterr3rrqrr)rrrrrs rrOz"AutoRegressiveCudaGraphKit.releases3m**,,  F NN     \((** ? ?E N # ' ' > > > >   ! + ( ( ( ( ( (  % %dk4: > > >%*D "- , ,Hx// ,'+$ ##%%%  &&(((((r )NN)r-r.rr)rr:rr)r)rrrr`rr.rr`)rr.r/r0r9r rr:r^) r!r"r#r$r8rrr[rOr%r rr}r}xs 3 3 3 3 3 & & & &,####,))))))r r})r)r_r`rra) __future__r dataclassesrtypingrrrrr'r,rgrvr}r%r rrs]""""""!!!!!!%%%%%%%%   ? ? ? ? ? ? ?  ?   77777777tSSSSS" ) ) ) ) ) ) ) )\)\)\)\)\)\)\)\)\)\)r