j6dZddlZddlmZgdZddddZejejfZd a d a da d a d a dad ad ad ad d ddZdZdZdZd!dZdZdZdZdZdZdZdZdZd"dZd!dZ Gdd Z!e!Z"dS)#aFP8 math kernels of FP8 checkpoints (Quanto qfloat8 weights and shared.qtypes.scaled_fp8 weights), chosen in the configuration (fp8_kernels): - pytorch: each weight is dequantized whole (copies of its size in 16 bits) before a 16-bit matmul; - triton: weight-only Triton matmul (fp8_w8a16_triton): FP8 weight tiles decoded in registers, no copy of the weight, same values as pytorch up to the order of the sums; - cuda: FP8 tensor cores (torch._scaled_mm, cuBLASLt, RTX 40xx and later): the activations are also quantized to FP8 (one scale per call), fastest, lower precision; - auto: the fastest available (cuda, then triton, then pytorch), as WanGP's FP8 checkpoints used FP8 tensor cores before. Block-scaled FP8 weights (shared.qtypes.block_scaled: one scale per 128 x 128 block, or per output channel and 128-element block of K) run on WGPKernels' FP8 GEMMs with E4M3 activations (quantized per row and 128 columns, approximate) under the cuda choice, else with 16-bit activations (exact, the weights decoded in registers): WGPKernels' W8A16 kernels when its library has them, else the Triton block kernel (shared.kernels.fp8_block_triton), on every GPU including those without FP8 tensor cores; without a precise kernel, the E4M3 kernels when the precision setting allows approximate kernels; the pytorch choice dequantizes them per call. A choice this GPU cannot run gives way to the next one (cuda -> triton -> pytorch), and the kernels in use are printed.N)is_fake))zAuto (fastest available)auto)zPyTorch (slowest, precise)pytorch)zTriton (fast, precise)triton)z!CUDA FP8 (fastest, least precise)cudaPyTorchTritonzCUDA FP8 Tensor Cores)rrrrFrzWGPKernels (E4M3 activations)zWGPKernels (16-bit activations)zTriton (16-bit activations))wgp_w8a8wgp_a16 triton_a16cd}|dkr@ddlm}||tjrdSd\}}|dkr* ddlm}d|fS#t$r}d d |d fcYd}~Sd}~wwxYwd |fS) zY(kernels this GPU runs for a pytorch / triton / cuda selection, reason when they differ).Nrr scaled_fp8)rN)rz.FP8 tensor cores need an RTX 40xx or later GPUrfp8_w8a16_tritonrzTriton is unavailable ()) shared.qtypesr_init_scaled_mm_support_scaled_mm_availabletorch float8_e4m3fnshared.kernelsr ImportError) selectionreasonrrexcs BC:\Users\johnn\OneDrive\Documents\AI\shared\kernels\fp8_backend.py _availabler#s FF,,,,,,**,,,  * *5+> ? ? <V 6H ? 7 7 7 7 7 7V# # ? ? ?>>>>> > > > > > > ? f s A A6$A1+A61A6c|dtDvrtd|t|dkrdn|\}}||f|fS)zy((kernels of the Quanto qfloat8 weights, kernels of the scaled FP8 weights), reason when they differ from the selection).ch|]\}}|Sr!).0_values r z"resolve_backend..7s77781e777zUnknown FP8 kernel backend: rr)CHOICES ValueErrorr)rresolvedrs rresolve_backendr*5sc77w77777E EEFFF!I,?,?&&YOOHf h  ''r&chddl} |d}|sdS|j|t j5|rdn|cdddS#1swxYwYdS#t$rYdSwxYw)z^wgpkernels.fp8 when installed with its library and its selftest passes on this GPU, else None.rNzwgpkernels.fp8) importlib import_moduledefault_library_pathis_file_lib load_libraryrinference_modeselftest Exception)r,modules r_probe_wgp_fp8r6=s (()9::**,,4466 4 ;     ! ! !  ! # # 9 9!??,,844& 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 tts;;B#.B#1B B#BB#BB## B10B1c |t|n|\\}}}|||ftttfkr tdz a|||caaa|dkrt ndat duott dda|dko ta ||fdkrtddl m }|t||kr t |nt |dt |d }t#}t%d ||rd t&|znd d ||rd|znd d||fS)Nr a16_supportedcdS)NFr!r!r&rzconfigure..SsSXr&)rrr)offload, z for scaled FP8 checkpointsz[FP8] Backend: z; block-scaled weights: z (setting: z; z).)r*backendscaled_backend _selectionrevisionr6_wgp_fp8getattr_wgp_a16_probe_triton_block _triton_block _register_opsmmgpr<register_row_projection_FP8_ROW_PROJECTION_LABELS _block_routeprint BLOCK_LABELS)rr)quantoscaledrr<kernelsblocks r configurerTLs=E=My999S[VVf "w &KKKA *0&)'G^Z#)Y#6#6~DHt#[(Y/==(Y(Y([([Hi'A,?,A,AM111 ##$7888!'6!1!1gfoo'&/7y7yWU[_7y7y7yG NNE bG bY^%f%?,uBU%U%Udf b bs| bPV^CFLLL\^ b b bccc 6>r&c6 ddlm}dS#t$rYdSwxYw)Nrfp8_block_tritonTF)rrWr4rVs rrFrF_s@333333t uus  ctdkrdSttdks tdkrdStrdStrdSddlm}t|rdndS) a0Kernels of the block-scaled FP8 weights for the active choice: 'wgp_w8a8' (WGPKernels, E4M3 activations) under the cuda choice, else the exact 16-bit-activation kernels 'wgp_a16' (WGPKernels) or 'triton_a16', else 'wgp_w8a8' when approximate kernels are allowed, else None (dequantized per call).rNrr r r r kernel_policy)r@rCrArErGrrZallow_approximaterYs rrMrMgs""t6!9!9Z6=Q=Qzy|,,,,,,!--2Q2Q2S2S-::Y]]r&cddlm}||jo;|jddzdko'|jddzdko|S)Nrrr8)rrrdtypeshape is_contiguous)weightxrs r _cuda_runsrcws((((((  * *6< 8 8 PV\!_r=QUV=V P[a[ghi[jmo[ost[t PyzNzNzPzPPr&cddlm}||dkr|dn||\}}||||tj ||nd|j }|X| ||j  dd||||S)zwFP8 tensor cores: x quantized to FP8 with one scale, the weight scale applied by cuBLASLt (one value) or to the output.rrr8N)rr_scaled_mm_weight_scalenumelreshape_scaled_mm_rows contiguousttorfloat32r^mul_viewadd_)rbrascalebiasrscale_b output_scaleouts r _cuda_linearrv|s((((((&>>TYT_T_TaTadeTeTeu}}R?P?P?PkprxyyG\  $ $Q\\^^VXXZZEMAZAZdpdx\`\`CEFEL M MC ++00B77888   HHTNNN Jr&c J|rXt||rt||||Sts/datd|jdt |jdddlm}| ||||| |jd|jdfS)NTz8[FP8] CUDA FP8 kernels cannot compute some layers (e.g. r=z): Triton computes them.rr) rcrv_cuda_skips_notedrNr^tupler_rrlinear new_empty)rbrarqrrrrs r _linear_implr|s L fa  865$77 7  L $  KV\KK]bcico]p]pKKK L L L//////  " "1feT1;;PQ TZT`abTcGd;e;e f ffr&c trdStjdddtjdtjdtjdtjdzdt d tjf d }|jd }d adS) Nzwan2gp_fp8::linearr!) mutates_argsrbrarqrrrreturnc(t|||||SN)r|rbrarqrrrs r linear_opz _register_ops..linear_opsAvudD999r&c\||jd|jdfS)Nr)r{r_rs rfakez_register_ops..fakes%{{AGAJ Q8999r&T)_ops_registeredrlibrary custom_opTensorbool register_fake)rrs rrHrHs ]1CC:U\:5<: :TYT`cgTg:os:x}yE:::DC::::OOOr&ctjst|r(tjj|||||St|||||S)zx (M, K) @ (weight (N, K) float8 * scale (1 or N values, in x's dtype))^T + bias with the selected kernels (FP8 tensor cores when cuda, else the Triton weight-only kernel), in x's dtype.)rcompiler is_compilingrops wan2gp_fp8rzr|rs rmatmulrs` ~""$$I Iy#**1feT4HHH 65$ 5 55r&c|jr:|jtjtjfvr|jr|jddzdksdSt S)zThe kernels (a BLOCK_LABELS key) computing a block-scaled FP8 weight for this input (BF16/FP16 CUDA activations, K a multiple of 16), or None for the per-call dequantization.rer]rN)is_cudar^rbfloat16float16r_rM)inputras r block_modersV Mekenem-LLLQWQ_Ldidoprdsvxdx|}d}d}t >>r&r cz|d|jd}|ddks/|ddzs|dzr|}|*|j|jkr||j}|dkrddlm}| ||||}nGtr#t |||||d k nt ||||}|jg|jdd|jdRS) ainput @ (weight E4M3 * scale per 128 x 128 block, or per channel and 128-element block of K).T + bias with the kernels of mode (block_mode()): WGPKernels with E4M3 activations ('wgp_w8a8') or 16-bit ones ('wgp_a16'), or the Triton block kernel ('triton_a16').rer8rr]Nr rVr )precise) rhr_stridedata_ptrrjr^rlrrWrzrErC)rrarqrrmoderbrWrus r block_linearrsG  b%+b/**Axx||qAHHQKK!Oqzz||b/@ LLNN DJ!'11wwqw |333333%%a==T\Jhooadi>OoPPPbjbqbqrsu{~CEIcJcJ 3; : CRC( :&,q/ : : ::r&c x|j|j}}|d|jd}|j|jkr||j}||j|jkr|nd}t ||||tdkjg|jdd|jdR}||||zn|S)zinput @ weight^T + bias for a Quanto qfloat8 weight with the selected kernels, computed as Quanto does in the dtype of the weight's scale.reNrr)_data_scalerhr_r^rlrr?)rrarrdatarqrb fused_biasrus rrzrzs, %D b%+b/**Aw%+ DD  )djEK.G.GTJ G&D%W-> ? ? G iUXVXUXIY i[_[efg[h i i iC)j.@3::cIr&cfeZdZedZedZedZedZdS)_FP8RowProjectioncxt|dd}| |jdvrdSt|s/tjst|dddS|j}|j|jkoE|j d|j dfvo|j dup|j j|jkS)N weight_qtype)qfloat8_e4m3fnqfloat8_e4m3fnuz qfloat8_e5m2scaled_float8_e4m3fnscaled_float8_e5m2Factivation_qtyper8r) rDnamerrrrqweightr^rrgr_rr)r5rbqtyperas rsupportsz_FP8RowProjection.supportss55 =EJ/SSS5 1:: 4466 '&J\^b:c:c:o5 'JFM,?,?,A,AaVWEY,YJ(HFK,=,H Kr&cddlm}|j}t||}|rtnt }|jsd}n+|dkr%|r|jst|j |j sd}d||j |j j fS)Nr)ScaledFP8WeightTensorrrrfp8_rows) shared.qtypes.scaled_fp8rr isinstancer@r?r_force_full_precision_mmrcrr^)r5rrarQrs rkeyz_FP8RowProjection.keysBBBBBBF$9::!'4~~W~ DD V^^&^V-L^U_`f`lntnzU{U{^D4v|/AAAr&cddlm}t|\}}}}||}|dkrg||||Rn|S)Nr)_quantize_activationr)rrrrrlrj)r5rbrr#rr^ fp8_dtypes rpreparez_FP8RowProjection.prepares{AAAAAA$5$9$9&$A$A!4 DDKK " " $ $?Cv~~;%%a33;U;;;STTr&cddlm}|j}|j||}|jdkr|jn!|jd||}|jdn|j||}t |d} | dkr6|\} } } ||z } |||\}}| dzrJ| | dzdzdz|j df}|d|  ||}|| tjkr | dzdkr|nd}tj| || |tj|| ddd| f}|<||| dd|||||S| d kr+t+||||j|d S||j||jddz}||z}||||S) Nr)rfr8rerr])rr out_dtyperF)rrfrrrrgrhrrrr new_zerosr_copy_rrm _scaled_mmrkrlrnrorprr^)r5preparedstartstoprfrarrqrrrrbxsr^countwsrtpaddedrrudenses rrowsz_FP8RowProjection.rowssDDDDDD|E$J'!'!4!4!6!6!!;!; AVAVWYAZAZ[`ae[eAf{*tt E$J0G $$V,,Q/ 6>>#LAr55LE66udCC B rz %"*);b)@$*Q-(PQQvv$$T***!-!5%5=:P:PUZ]_U_cdUdUdjnJ"1dffhhBEE%-4H4HzejkkklmlmlmoupuouluvC'//44Q;;<<<J$6J 8  (D%((8>*B*BD%PP P''%((8>*B*B*J*J2q*Q*QQ"   HHTNNN r&N)__name__ __module__ __qualname__ staticmethodrrrrr!r&rrrsKK\K B B\ BUU\U \r&rr)Nr )#__doc__rtorch._subclasses.fake_tensorrr'rLr float8_e5m2 FP8_DTYPESr?r@rBrrxrCrErGrArOrr*r6rTrFrMrcrvr|rHrrrrzrrKr!r&rrs { { 111111 : : :8=T U U!5#4 5      ;HizWXX $(((   & ^ ^ ^ PPP    g g g    666 ; ; ; ; J J J J========@('))r&