j:dZddlZddlZddlZddlmZddlmZddl m Z iZ dZ dZ dad ZGd d Zd ZGd dZGddeZdS)zShared SwiGLU expert execution. Architecture code supplies routing, not residency, and quantization handlers supply the storage formats and their kernels (ExpertFormat).N) functional) ExpertBank) allocatorc||tvr'tj|t|<t|S)zEThe stream of the GPU expert branch of queue_forward, one per device.) _side_streamstorchcudaStreamdevices ;C:\Users\johnn\OneDrive\Documents\AI\mmgp\expert_compute.py _side_streamrs2 ]"" % 1 1& 9 9 f   c ttjdt dt jS)NMMGP_CPU_EXPERT_THREADS)intosenvirongetminr get_num_threadsrr cpu_threadsrs4 rz~~7Q@U@W@W9X9XYY Z ZZrcntptjtjdkrdnd}tjtjtjg|j_tj|j_ |jat|j |tj |}|rtd|ddS)aQueue function(&job) on a CUDA stream (captured as a host node inside a graph capture): it runs when the work queued before it is done, and the work queued after it waits for it. The job (a ctypes structure) must stay alive while it may run or be replayed.Nntz nvcuda.dllz libcuda.so.1zcuLaunchHostFunc failed ()) _host_launchctypesCDLLrnamec_void_pcuLaunchHostFuncargtypesc_intrestype cuda_stream addressof RuntimeError)functionjobstreamdriverstatuss rlaunch_host_jobr0sRW__\\.QQ,2OV_fo+^(*0,'. &,h8H8M8M N NF B@v@@@AAABBrc<eZdZdZdZdZdZdZdZd dZ d Z dS) ExpertFormata{Storage and kernels of routed SwiGLU expert banks in one checkpoint format. Quantization handlers supply formats through their EXPERT_FORMATS (mmgp.quant_router.expert_formats()); a bank takes the format of the checkpoint tensors it loads. Every tensor holds one entry per expert along its first dimension. bank.rotation is the ConvRot group of the activations (0: none).ct)ziCheckpoint name -> (shape, dtype) of a bank's tensors, in the order of the rows of the GPU pointer table.NotImplementedError)selfexpertshidden intermediates rtensorszExpertFormat.tensors3!!rct)zgWhether a bank's checkpoint tensors (name -> tensor, the names of tensors()) are stored in this format.r5)r7r;s rmatcheszExpertFormat.matches7r<rct)aR(job, host function address) of a CPU computation from the addresses of a pinned routing payload (bank.pack()), of resident flags (or 0) and of a pinned float32 [tokens, hidden] result (zeros when every pair is computed elsewhere). The job is a ctypes structure passed by address; a nonzero status field reports a failure.r5)r7bankpayloadresidentouttokenstopkthreadss rcpu_jobzExpertFormat.cpu_job;s "!rc t)zThe same computation from the addresses of float32 activations and probabilities and int32 ids: returns the number of pairs computed (out is left unwritten when there are none).r5) r7r@xids probabilitiesrBrCrDrErFs r cpu_forwardzExpertFormat.cpu_forwardA "!rNct)afloat32 [tokens, hidden] sum of the selected experts having GPU copies (pointers: int64 [experts, tensors] table of their addresses, zero rows contributing nothing); out and the float32 [tokens * topk, intermediate] scratch mid may be preallocated.r5)r7r@rIrJrKpointersrCmids r gpu_forwardzExpertFormat.gpu_forwardFrMrct)zfloat32 gate and up [2 * intermediate, hidden] and down [hidden, intermediate] weights of one expert (name -> tensor), as applied to unrotated activations.r5)r7r@r;s r dequantizezExpertFormat.dequantizeKrMr)NN) __name__ __module__ __qualname____doc__r"r;r>rGrLrQrSrrrr2r2-s~~ D""""""""" """ """" """""rr2c6ddlm}|S)Nr quant_router)r3rZexpert_formatsrYs r_formatsr\Qs&  & & ( ((rcJeZdZdZdxZxZZedZedZ dS) _HostStagingaPinned buffers shared by the banks of the process, which run one after the other: the routing payload comes back in one blocking copy, the CPU result goes up from pinned memory. `uploaded` orders the next write of the result buffer after its transfer.Nct||}|||krPtjt |d|dn|z|dd}t ||||S)NrcpuTdtyper pin_memory)getattrnumelr emptymaxsetattr)clsr"rfrccurrents rbufferz_HostStaging.buffer\s#t$$ ?gmmoo55k#eQw!!GMMOO-\"]"]ejsxFJKKKG Cw ' ' 'rcp|j|jdx|_x|_|_dSN)uploaded synchronizeinputsoutputs)rjs rreleasez_HostStaging.releaseds7 < # L $ $ & & &266 6S[3<<z)SwiGLUExperts.__init__..ws0!d!d 9>]cKcKc)KcKcKcKc!d!dr) super__init__r9r:rxrwrotation_pointer_table_pointer_version_jobs _set_formatnextr\)r7 num_expertsr9r:rwr rx __class__s ` rrzSwiGLUExperts.__init__ns b))))/& T&$%," T]" "     T!d!d!d!dXZZ!d!d!dddfl m m m m m  rc >|jD] }|j|= ||j|j|j}|D]3\}\}}||tj |||4t||c|_|_ dS)Nrcr ) expert_tensor_names_buffersr;rr9r:itemsregister_bufferr rgtuplerw)r7rwr r"declaredshapercs rrzSwiGLUExperts._set_formatys, $ $D d##>>$"2DKARSS$,NN$4$4 W W D.5%  u{5f'U'U'U V V V V05h- $+++rc <dzdx}&t||_t D]} | |j|j|j} tfd| DrC| fd| Dr!| |j ur| | dntdddt|||||dS)Nconvrot_group_sizec3&K|] }|zvV dSrnrr|r"prefix state_dicts rr~z6SwiGLUExperts._load_from_state_dict..s,AA46D=J.AAAAAArc(i|]}||zSrrrs r z7SwiGLUExperts._load_from_state_dict..sBVIVIVIz~VZ\fgmptgt\uVIVIVIrmetaz2No registered expert format stores the tensors of rz)popritemrr\r;rr9r:allr>rwrr*r_load_from_state_dict) r7rrlocal_metadatastrict missing_keysunexpected_keys error_msgsgrouprwnamesrs `` rrz#SwiGLUExperts._load_from_state_dictsg^^F-A$A4HH HE U --DMjj c cFNN4#3T[$BSTTEAAAAA5AAAAA fnnVIVIVIVIVICHVIVIVIGJGJ ,,$$VV444aTZ[^\^[^T_aabb b %%j&.&R^`oq{|||||rcd|_d|_t|jr9t j|jdSdS)Nrz) rrr^rsrr r rpclear)r7s rrelease_execution_cachez%SwiGLUExperts.release_execution_cachesf" " :  J " " $ $ $ J         rc |j}|d|j}|j\}}|||||}||jvrt j|t j dd}t j||jt j dd} |j || |jr|j nd| ||t!\} } | | || f|j|<|j|\} } }} t j} |jot'|jx} r||jt/|j}}||}}t j||jt j |j}t j||z|jt j |j}|| t j|5|j |||||||dddn #1swxYwY||d t;| | | t j| jt j |j| d }| r*| |||||j |S) aBGraph-safe forward on the current CUDA stream: the CPU computes the selected experts in a host function queued between the copy of the routing payload to pinned memory and the copy of its result back, so a captured graph replays the whole call without returning to Python. With the Triton backend, the experts having GPU copies are computed on the GPU instead, in a branch running beside the CPU job; a placement change releases the graphs that read these copies. Failures are reported by raise_job_errors() once the stream has been synchronized.rzNraTrbrr)rCrP non_blockingrc)!rreshaper9 contiguouspackrr rgrfint32float32rwrGdata_ptrrxresident_flagsrr current_streambool _resident _pointersr rr: wait_streamr-rQcopy_r0add_torc)r7rItopk_ids topk_weightspackedrrDrErAresultr,r+r-gpurOside gpu_outputrPoutputs r queue_forwardzSwiGLUExperts.queue_forwards  IIb$+ & & 1 1 3 3~  >YYq(L99F  # #k&,,.. E^bcccG[EMRWdhiiiF K//g6F6F6H6Hlpl{KC$J]JfJfJhJhJhBCEKETETEVEVX^`dfqfsfsttMC!$h!?DJv )-F);&Xw**,,/:d4>&:&: :3 l!^^AH55|AH7M7MdH%-%8%8%:%:Lem\]\deeeC   V $ $ $""4(( l l ''a<Wagj'kkk l l l l l l l l l l l l l l l f4 000#v...V\qxPPPVVW]lpVqq  $   t $ $ $ KK # # #yyqwy''//666s1"JJ#&J#c|jD]0\}}}}|jr"|jdc}|_td|d1dS)z`After a stream synchronization: raise the failure of a host job queued by queue_forward, if any.rzCPU expert kernel failed (rN)rvaluesr/r*)r7r,_r/s rraise_job_errorszSwiGLUExperts.raise_job_errorssp J--// K KLCAqz K%(Z" "#I#I#I#IJJJ K K KrcjjksjfdtjD}t jrtdtj 5t j |t j |_dddn #1swxYwYj_jS)Ncg|]I}j|xrfdjDndgtjzJS)cDg|]}|Sr)r)r|r"rBs r z6SwiGLUExperts._pointers...s)TTT4Xd^,,..TTTrr)rrrlen)r|expertrBr7s rrz+SwiGLUExperts._pointers..s;;;fjesewewx~eeYYajTTTT4;STTTTHIGJMPQUQiMjMjGj;;;rzBThe GPU expert pointer table must be prepared before graph capturer) rresidency_versionrrangerr r is_current_stream_capturingr*rnormal_allocationstensorint64)r7r rowsrBs` @rrzSwiGLUExperts._pointerss   D$: : :d>Q>Y;;;;;"'(8"9"9;;;Dz5577 i"#ghhh-// [ [&+l4u{SY&Z&Z&Z# [ [ [ [ [ [ [ [ [ [ [ [ [ [ [$($:D !""s;'B..B25B2ctj|dtj|dtj|dtjfS)zforward()'s routing payload as one int32 tensor: float32 activations, float32 probabilities, int32 ids. Graph-safe: computing it in the graphed stage that routes saves its launches.rz)r catrfloatviewrr)rIrrs rrzSwiGLUExperts.packsy!))B----//44U[AAYYq(L99Fx}&  &&xMMD &,,.. ! ' ' / / / /D% 3Vt{T?Q5R*:j6D=889??AA ZZ\\$ ;XJ iDK,0022 ij$%hiii? i!%!8!8q||~~xObObOdOdfrf}f}ffBFBPBPQRQYBZBZ"["[JJ"'!1!5=!I!I!IJ#.s3388FFG"(SZZ\\):):)G)G)Q)Q"R"Rii&+k'V2C&D&D e&*ggah&7&7!(9K9Ke"&+"8"8x?O"P"PR#$8AdGMMOOS#A#A#G#Gr#G#R#Rb!"!&,,*;R!@!@"--av TSXZ^M^@_@e@e@g@g7ghhhh R%,,Y%-PP(4 )55777XU]5QQQ==??DK++D$q:~8MtVWZbVbObdhdweAeAeCeCEKETETEVEVX^`dfqfsfsttE l  *// DD3"-fQXFFFLLVbfLggF,8,A,WUZEUEUEWEWL) )00222)KK +++'1'=5CSTU]b]jCkCkCk99179++33E::I$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;sQU**U.1U.rn)rTrUrVrWrrrrrrr staticmethodrr inference_moder __classcell__)rs@rrvrvksDDDHQVch n n n n n n nHHH } } } } }!7!7!7!7FKKK # # #^^\^ U2;2;2;2;2;2;2;2;rrv)rWr rr torch.nnrrr8rr3rrrrrr0r2r\r^rvrrrrs?-- $$$$$$ !!![[[  B B B!"!"!"!"!"!"!"!"H))) 77777777*Y;Y;Y;Y;Y;JY;Y;Y;Y;Y;r