j<dZddlZddlZddlZddlmZddlmZddl m Z e dddZ dZ ej Zd Zd Zd Zd,d ZejgddejdejdejdejdejdejdejdejdejdejdejdejfdZdZejdgdejdejd ejfd!Zd"Zd#Zd-d$Zd%Zejgd&dejdejd'ejdejd(ejdejdejfd)Zd,d*Zdd+lmZedS).a~Triton implicit-GEMM 3D / 2D convolution on channels-last 16-bit activations ("fast" kernel precision), shared by the Wan VAEs and the streamed LTX VAE: FP32 accumulation, the bias and an optional residual added as PyTorch rounds them, optionally the nearest x2 upsampling of the input read in place (Wan decoder). 1.1-2x faster than cuDNN on RTX 50 GPUs, where it is enabled; only the accumulation order differs. One binary per channel count, kernel shape, mode and dtype; sizes and paddings are not specialized. Launch configurations start from a static choice and are tuned once per identity and GPU on scratch outputs of a large call.N)dispatch) LaunchTunerwan2gp_conv3d_tritonVAEicdtjtj|tjS)zNVIDIA GPUs with compute capability 8.0+ (the 16-bit kernels' floor, shared.kernels.dispatch); other devices use the PyTorch path.)rdevice_supportedtorchdevicebfloat16r s DC:\Users\johnn\OneDrive\Documents\AI\shared\kernels\conv3d_triton.py supportedrs!  $U\&%9%95> J JJct|o4tjtj|jddkS)zTriton convolutions: measured faster than cuDNN on compute capability 12.x GPUs only (GeForce RTX 50; 0.7-1.3x slower on 8.9), so other GPUs keep cuDNN. A performance choice, not an eligibility one.r )rr capabilityr r indexr s rconv_supportedrs; V   Y!4U\&5I5I5O!P!PQR!SWY!YYri Fc*|jjdd\}}|r-|jd|jdz|jdzdz}nj|jd|jd|jdz dzz|jd|jdz dzz|jd|jdz dzz}|jtjkoN|dzdkoE|dzdkoz conv_eligible..*s*PmPm\]QRVWQWPmPmPmPmPmPmr) weightshape kernel_sizedtyper float32allstride_CONV_MIN_PIXELSrr )xconvupsamplecoutcinpixelss r conv_eligibler.#sk !"1"%ID#Wagaj(171:59qwqzD,r?r@rArBBNpidnum_mnum_nfirst_msize_mpid_mpid_nmrowswrhtbn1n2kacc1acc2khuhkwuwinsiderowk0aweightsbasektr#ys9 r _conv_kernelr}-sSyB -  C GAw  E"*Ew'72G Z 1 1F sgo.&8 8E GeO $ /E W$ryG'<'<,>,A,A"*,M,MdTUTUTUg,V V  GU rz**RWSAHaaag;N5NQSTXZ[Z[Z[T[Q\5\^bcdcdcdfjcj^ksv-w-w-w-z-z{}|F.G.GGA a$h4((2dAAAg;6U T!!!T']SSSSS wri@TAPSBLOCK_CBLOCK_Tc  tjdtj|z|z}tjd|ztjd|z}tjd|}||kdddf||kdddfz} tj||z||zdddfz|dddfztj||z||zdddfz|dddfz| | dS)zBLOCK_C input channels of one output channel of a convolution weight (cout, cin, taps) written channels last (cout, taps, cin).rrN)rIrJrMrNrOrWrS) r2OUTr7r~rrrzcrjmasks r_weight_kernelrts =    rx ( (3 . 5D a7"RYq'%:%::A !WA GQQQW T47 3 3DHS4Z1s7D!!!G, ,qDz 9271t8qSWxYZYZYZ\`Y`NaCadefjlmlmlmfmdnCnpt;u;uw{|||||rc v|d}|dkrtn tj}|dksK|dks3|t ks||r||Stj||}t|j dtj |j ddf|||j d|dtj |d|S) aweight (cout, cin, *kernel) in the channels-last layout of the channels-last convolutions: large 3D weights by a tiled transpose, 2.5x faster than PyTorch's strided copy (tiled decodes convert every weight once per tile and chunk); small ones are launch-bound.)rrr memory_formatr@r) num_warps)numeldim_CLr channels_last_WEIGHT_KERNEL_ELEMENTS is_contiguous contiguous empty_likerr!tritonrKnext_power_of_2)r tapslayoutouts rchannels_last_weightr~s8 $<    DJJLLA%%SS5+>F qyyFJJLLA%%:Q)Q)QU[UiUix~UiUU)Q  v 666  6 8 8 8CFLOV[a"%E%EFGHYHYH[H[]`bhbnopbqswy{~D~TUY~Z~Zfghhhh Jrc|dzdkrdn|dzdkrdn |dzdkrdnd}d g|d d d Rd g|d d d Rdg|d d d Rg}|d zdkr|d t|S)zLaunch configurations (BLOCK_M, BN1, BN2, BLOCK_K, warps, stages), the first being the static choice: output channels by 64 (96 = 64 + 32: no power-of-two padding), and by 128 when they divide.rr)rr`)rr)r)rrrrrr)rrrrrr)appendtuple)r+split candidatess r_conv_candidatesrsbyA~~GGtbyA~~88X\_bXbfgXgXg99mtE)))A)q))C+B%+B+BQ+B+B+BRDZ%DZQSDZUVDZXYDZDZ[J czQ1222   rc |jdd\ r tjntdntrTj\}}ddddddzdzf\ tj| fjjtj}nej\}}|jdd\z dzz dzz dzc tj|  fjjt}|z zzt|f fd  fd} d d d d d rd nd d d ndd j} t } |t d| j| | ztk|S)afConvolution (stride 1) of the padded channels-last input x (batch, cin, TP, HP, WP) by weight (cout, cin, kt, kh, kw), or with upsample of the nearest x2 upsampling of x (frames, cin, h, w) by a 3x3 weight with zero padding 1, plus bias (and residual, the output's shape and layout), as the PyTorch path rounds them. Returns the channels-last output.Nrrrrr#r rc|\}}}}}} ttj|| ||zzzf ||n|||  |du||||d|| dSN)r num_stages)r}rrK)targetcountconfigresblock_mbn1bn2block_kwarpsstagesbiasr,r+rihprqr{rsrjtpr*rgwpwtr(s rlaunchzconv..launchs4:1c7E6fk%11TcCi5HIKLQPRTXdgdoZ`Z`ux{ACHJKMNPQMOQSUWY\^bdfhjlnpxz}FJ{JMTVY[^`gijv{IO P P P P P PrctdtjzjjfdS)Nr#r c |SNr)rrrresidualrs rz'conv..scratch..sffVUFHEErminr emptyr#r )rrr+rr-rr(s@@rscratchzconv..scratchsJFG$$UT\JJJEEEEEEEEr|r(r*paddedrr)) r!rr rrrr#r rr_TUNERr_TUNE_ELEMENTS)r(r rrr*batch_rridentityrr,r+rirrqr{rsrr-rjrrgrrs` ``` @@@@@@@@@@@@@@rr)r)s RaR ID# (#K5#6#6 LLA'ttX-@-@s-@-S-SHe7q"b"#Q1aRR"?BB1ak5$1-QWQX]b]pqqq wq"b"\!""% Br'A+rBw{BGaK1ak5$1a0`cddd QY]Q F f % %B*2PPPPPPPPPPPPPPPPPPPP FFFFFFFFF LLLLrLLBLLLLH.Rjj(LLckcwU_U_}LLCDCJLLH!$''J F3 fh*gW]`dWdhvWvwwxxx Jrc|jtjko|jo|jdko|jdko|jdko|jdkow|jdkol|j dkoa|j dzdkoS|j dzdkoE|j d|j dz|j d ztkot|jS) zTriton 2D convolution for conv (3x3, padding 1 with zeros, stride 1, input channels multiple of 32, output channels of 64) on the 16-bit input x (batch, cin, h, w) of a validated GPU.)rr)rrrrPrrrrr)r#r r$is_cudar"paddingr&dilationgroups padding_mode in_channels out_channelsr!r'rr )r(r)s rconv2d_eligiblers  Gu} $ F Ft7G67Q FVZVbflVl Fquq|AGrG F ' F,0K1,< FAEARV]A] Fbfbruwbw{|b| F!B&!+ F01 QWQZ0G!'RS*0TXh0h Fm{|}}EnFnFGr)r/r1r2RELUr]c>tjd}tj|| }|| z}|| |zz| z}tj||z | }||| |zz|zz}|| |zz|z}|tj| ztjd| z}||k}||z}||z}||z}||z}|| ztjd| z}tjd| }tj| | ftj}tj dD]}||zdz }tj dD]} || zdz }!||dkz||kz|!dkz|!|kz}"||z|z|z|!z|z}#td|| D]}$tj ||#dddfz|$|zdddfz|"dddfd}%tj |%tj tj ||dz|zdddfz|dz| z|z|$z|zdddfz|}|jj}&||&tjtj ||ztjdddfz|&}'| rtj|'d}'tj|||zdddfz|dddfz|'|dddfdS) a Implicit-GEMM 3x3 convolution, zero padding 1, FP32 accumulation, of BLOCK_M output pixels x BN output channels: X channels last (batch, H, W, CIN), WT (COUT, 3, 3, CIN). As PyTorch: the convolution rounded to the output dtype, plus the bias, rounded, then ReLU.rrDrrNrErFrH)rIrJrKrLrMrNrOrPr$rQrRrSrTrUr#rVmaximumrW)(rXrYrZr\r/r1r2r7r8rr>r]rArBr^r_r`rarbrcrdrerfrgrhrirknrnaccrqrrrsrtrurvrwrxr#r|s( r_conv2d_kernelrsm -  C GAw  E"*Ew'72G Z 1 1F sgo.&8 8E GeO $ /E W$ryG'<'<.launchs.4+WeV E733trzBDEaTSY[`bcefhkmqswzACEGNPQOTag i i i i i irc~tdtjzjjfdS)Nrrc|Srr)rrrrs rrz)conv2d..scratch..sffVUF;;rr)rrr+rr-r(s@@rrzconv2d..scratchsGFG$$UT\JJJ;;;;;;;r))rrrrr)rrrrr)rrrrrr))rrrrrrrz|3x3|rrconv2d) r!rr rrPr#r rrrr)r(r rrrrrrrr,r+rirr-rgrs` `` @@@@@@@rrrswE31 <?D 5#6 77A@D 5;t1718 < < < rs dd  ######333333 +Q 6 6 KKK ZZZ jjjj DDDEEE@T2<@TWYWc@Tikiu@T{}|H@T\@T-/\@TEG\@T\^\h@Toqo{@Tl@T-/\@TDFL@T@T@TFE@TF" ug&&&}bl}R\}TVT`}}}'&}   !!!!PGGG ooo...!E!ER\!EY[Ye!Eprp|!E|!E.0l!EEG\!E!E!E/.!EH4TSSSSS!!#####r