j?dZddlmZddlZddlmZ ddlmZddlm Z dZ n#e $rddl mZddl m Z dZ YnwxYw d d lm ZmZn#e $rdxZZYnwxYwd d iZd d iZdZddd dZdZdZdZd+dZdZeddZeddZeddZdZdZdZ d,dZ!ej"j#dZ$d,dZ%ej"j#d Z&ej"j#d,d!Z'ej"j#d-d"Z(d#Z)d$Z*d%Z+d&Z,d'Z-d(Z.d)Z/ej"j#d.d*Z0dS)/aSageAttention (1): INT8 q/k per block of tokens and fp16 v, the Triton kernels of sageattn_varlen (SageAttention 1.0.6, or the copy SageAttention 2 keeps in its triton package), launched without its device sync (the sizes of its scales read on the GPU) so that CUDA graphs can record them, and in stages for the attention kit: v, k and q quantized as soon as each is computed, the output written into q. Same values as sageattn_varlen for sequences of the same length (all batch items), head_dim 64 or 128, no mask.) lru_cacheN) token_mean)quant_per_block_int8_kernel) _attn_fwdcd|dzz S)N?g?head_dims 9C:\Users\johnn\OneDrive\Documents\AI\shared\sage1_core.py _sm_scaler sX_$$c |dzS)Ngr r s r r r s 4r)r quant_per_block_int8_bits_kernelscale_vFT)@g@ic Ttj|td|d}tjtjtj|dtz  dS)zuPowers of two, at least 1, per head and channel of v (its batch and token dimensions: tokens_dims, kept with size 1).infT)dimkeepdimr)min) torchlinalg vector_normfloatexp2ceillog2mul__V_LIMITclamp_)v tokens_dimsamaxs r v_scalesr*(sx < # #AuU||d # S S Y Y [ [D :ejDIIa(l,C,C!D!DEE F F M MRS M T TTrrrc |s!|tjdfSt||}tj||tj|jtj|j|fS)zKfp16 v and, with scale_v, its v_scales (v divided by them), None otherwise.Ndtypedevice)out)torfloat16r*divemptyshaper/)r'rr(scales r fp16_vr7.sh )ttEM""D(( Q $ $E 9Q5;qwemTUT\#]#]#] ^ ^ ^`e eerc|j||j|z}||dkr|n|||S)zkout multiplied in place by the v scales of its attention (out has more heads with grouped query attention).r)r5r$repeat_interleave)r0r6 heads_dimgroupss r scale_outputr<6sH Yy !U[%; ;F 88Vq[[EEe.E.Efi.X.X Y YYrcTtj|ddkS)Nr)rcudaget_device_capabilityr/s r _fp16_requiredrB<s" : + +F 3 3A 6 ::rc2tduot|S)zbAttention masks run on the Turing kernel (compute capability 7.5); the packaged kernel takes none.N)_attn_fwd_bitsrBrAs r masks_supportedrEAs  % @.*@*@@rcLt|rtddntS)zHead dims of the kernels on this device: at 256, q blocks of 128 rows need more shared memory than before compute capability 8.0.N)rB HEAD_DIMSrAs r head_dimsrIGs#+622 A9RaR== ArcVtfdt|DdS)aKThe head_dim the kernels run a head_dim at: the next of head_dims(device), None beyond. attention pads q, k and v with zeros up to it, as SageAttention 2 does: the same q.k products (INT8, summed exactly), INT8 scales and key means, with the softmax scale of the real head_dim; the extra output channels (zero) are dropped.c3(K|] }|k|V dSNr ).0rr s r z"padded_head_dim..Qs'EESH______EErN)nextrI)r r/s` r padded_head_dimrPMs2 EEEE & 1 1EEEt L LLrcTtjd|dz|z|tj|S)zr[0, length, 2 length, ...] of the batch items, made on the GPU (a host copy would not be allowed in a CUDA graph).rrr-)rarangeint32)batchlengthr/s r _offsetsrVTs) <EAI/u{SY Z Z ZZrcftduo(|jtjkot |jS)zdA BF16 tensor on compute capability 7.5: the Turing kernels read it as bits instead of an FP16 copy.N) _quant_bitsr.rbfloat16rBr/)xs r _bits_inputr[Ys, d " ]qw%.'@ ]^TUT\E]E]]rc|j\}}}}||zdz |z}tj||z||ftj|j} tj||z|ftj|j} t |||jt |||j} } t|rt|||f| tj | | || n|| | | d| d| d| d|||||du| | | | fS|tdt|||f|| | | | | d| d| d| d|||| | | | | fS) a8x (batch, tokens, heads, head_dim), batch items one after another in memory, as INT8 rows (batch * tokens, heads, head_dim) and scales per block of tokens and head, with the offsets of the batch items in both. mean (heads, head_dim), FP32: subtracted first inside the Turing kernel (BF16 x read as bits).rr-NrGr)sm_scaleHCBLKHAS_MEANzFthe key mean is folded into the quantization on the Turing kernel only)r]r^r_r`)r5rr4int8r/float32rVr[rXviewint16stride ValueErrorr) rZblockr]meanrTtokensheadsr blocksx_int8r6offsets scale_offsetss r _quantizerp^s2&'W"E65(unq U *F [%&.%:%*UVU] ^ ^ ^F K/u}QX V V VE%eVQX>>PVXYX`@a@a]G1~~5VUE*+AFF5;,?,?Z^ZfPVPVlpry|I,-HHQKK!fmmTUFVFVX^XeXefgXhXh5=(X]hltxhx z z z zug}44 abbb 6765'S`bcbjbjklbmbmopowowxyozoz}C}J}JKL}M}MOUO\O\]^O_O_AIUV^dikkkk 5'= 00rc|d}|tL|jtjkr7|ds/|jdt krt|jr|ddfSgt||ddRS)zfp16 v with its scales (fp16_v, None without settings["scale_v"]); v is released. On compute capability 7.5, an unscaled BF16 v of a short sequence stays as it is: the Turing kernel reads its bits (staged_attention).rNrr) clearrDr.rrYr5_V_BITS_MAX_TOKENSrBr/r7)v_listsettingsr's r staged_quantize_vrvss q A LLNNN"qw%.'@'@R[I\'@abahijakpBbBbBqx((bB$} 0F1hy) * * 0D 0 00rc <|jdg|jddR}t|r,t|dd|t |dddn|S|t d |jtjkrNt|j r:| tj }|jdg|jddR}t |d}t|r| |n||z |jddS) zINT8 k minus its mean over the tokens of all batch items (or mean, (heads, head_dim) FP32: Turing kernel only), with its scales.rGNrrrT)fp32z8A given key mean is subtracted by the Turing kernel only)rdr5r[rprrgr.rrYrBr/r1r2sub_)kin_placerirowss r _quantize_kr~s 16" #qwqrr{ # # #D1~~bB$,Zad%C%C%CA%F%F\`aaa STTTw%.  ^AH%=%= DD  qvb'17122;''' dA  D Bdiioootd{HHQQSUWZ [ [[rc|d}|t|d|dS)aINT8 k minus its mean over the tokens of all batch items, with its scales; k is released (the mean is subtracted in place). settings["k_mean"] (heads, head_dim) FP32, when present, is the mean subtracted instead (a cached prefix's: shared.attention_kit).rTk_mean)rrr~get)k_listrur{s r staged_quantize_krs8 q A LLNNN q$ X 6 6 7 77rc|jtjkrBt|jr.t |s|tj}t|d|t|j dn|dzS)zINT8 q with its scales (the softmax scale, in base 2, goes into them: sm_scale, or that of q's head_dim); q is not released (staged_attention can write the output into it).rNrxg/ldG?) r.rrYrBr/r[r1r2rpr r5)qrur]s r staged_quantize_qrsq w%.  ^AH%=%= kRSnn DD   Q8Hi 444hZdd e eerc |\\}}}}\} } } } \} }}||j\}}}}|d||| jdg| jddR} }|jtjkot duot|j}| jtjk}|r|std|j|jtjkot|jo| }|r+t j |jtj |j}|dzdz||f}|| |r| tj n| || || || |r|tj n|f }| d| d | d| d | d| d | d| d f}tdd |d|d krd nd t| }|rd\}}||jtjkr|tjn|tj}||||| jd |z}|jtjkrdnd}|r)t j |||ftj|jn|}t |g|||n||||| nd||| jdzR|||d|n<||rtdt+|g||||| jdzRi||r-|||j||nt/||d}|r||fn|S)aAttention of the staged [q, k, v], written into out and returned: out has q's shape (batch, tokens, heads, head_dim) and its batch items one after another in memory: q itself, or a slice of heads of a larger output. Fewer heads in k/v: grouped query attention. return_lse (Turing kernel only): (out, lse), lse (batch, heads, tokens) FP32 the base-2 log-sum-exp of each query's scores without the keys' smoothing mean.rxrGNzFSageAttention on this GPU takes a BF16 v only with a BF16 output, not r-rrrrrr>)BLOCK_MBLOCK_NHEAD_DIMSTAGE num_warps num_stages)Nr)rrrr)V_BITS MASK_MODE RETURN_LSEzZSageAttention takes an attention mask or returns the log-sum-exp only on the Turing kernel)rrr5rdr.rrYrDrBr/rgr4r2rerfdict_STAGESbooluint8r1rcexpandrcopy_r<)r0quantized_listru attn_mask return_lseq_int8q_scale q_offsetsq_scale_offsetsk_int8k_scale k_offsetsk_scale_offsetsr'v_scale_rTrjrkr r}bitsv_bits fp16_outputgridargsstrideslaunchmask mask_modelses r staged_attentionrs uCq1VWi3`FGYP_bqcdfmop%(Y"E65(hhr5(++VQVB-E-E-E-E!D 9 & d>+E d.Y\YcJdJdD W &F odombebkmmnnn)u~-Y.2L2LYUYQYKO{4:U]3:NNN c\c !5% 0D F6@AFF5;///q)YX_ahjy|K&* 4DIIek " " " 6D}}Qq!1!16==3C3CV]]STEUEUWXW_W_`aWbWbdedldlmndodoquq|q|}~qqBFBMBMNOBPBPQG #rHAV^bdVdVdQRQRjkxAIyJKKKF T!i  2;/UZ2O2O9>>%+...U^UaUabgboUpUpD;;ueVV\!_5MNND&_ ::I]gsek5%0 cjYYYYmst Cd CdlFF Cc CG ChlhxW[WbWbWdWdWdK C" C$)V\!_$< C CEKW`mw C C|B C C C C  J yzz z$SSSSa0HSSSFSSS( $))CI&&'''##l3&C&CC# ,C::,rcRtduo|tjkot|SrL)rDrrYrB)rur/r.s r prefix_supportedrs'  % \%5>*A \nU[F\F\\rc|SrLr )rur/rs r prefix_settingsrs OrcdSNFr )rus r prefix_fp8_vrs 5rct|jdg|jddRddS)zs(heads, head_dim) FP32 mean over the tokens of all batch items, rounded to k's dtype as SageAttention 2's prefixes.rxrGNr)rreshaper5r )r{s r prefix_meanrsA iai1QWQRR[1111 5 5a 8 > > @ @@rc*||j|jSrL)startstop)rirks r prefix_mean_headsrs  EJ& ''rc|djdd\}}t|i|d|i\}}}}|j||g|jddR||d|jdfS)zA prefix's k (batch, tokens, heads, head_dim, released) quantized with mean: k_int8 (batch, tokens, heads, head_dim) and k_scale (batch, blocks, heads).rNrGrrrx)r5rrd)rrurirTrjrrrs r prefix_quantize_krs1IOBQB'ME6-f6R6R8T6R6RSSFGQ 6;uf 8v|ABB'7 8 8 8',,ubRYR_`bRc:d:d ddrc6|dddd|j|jf|dddd|j|jf}}|jd|kr/||ddd||dd}}|jd|jd}}|jdg|jddR|d|jdt |||jt |||jfS)aThe quantized k of staged_attention for the heads of a cached prefix: k_int8 (batch, tokens, heads, head_dim) and k_scale (batch, blocks, heads) as _quantize makes them (batch items one after another), for batch items (one prefix item serves them all).NrrxrrG)rrr5rr contiguousrVr/)rrrkrTrjrls r prefix_krsQQQ5;uz#99:GAAAqqq%+V[V`J`D` >PRT[TabdTe@f@f@q@q@s@s UFFM 2 2HUFFM4Z4Z \\rc |\}}}||jd t |j|t n|c } kr fd|||fD\}}}|rt nt }dt|dt|g|g} dx}}t|||| d<t|r|ntj |tj | ||} kr| n | dd fS)aRsageattn_varlen of q, k and v (batch, tokens, heads, head_dim, batch items one after another in memory) handed off in qkv_list, every batch item with all its tokens. k and v are not modified (others may hold them); the output is written into q with recycle_q. scale_v: v scaled by powers of two (fp16_v). attention_mask: SageAttention's (batch, heads, q, k) layout (shared.attention.sage_attention_mask), Turing kernel only (masks_supported). sm_scale: the softmax scale, that of the head_dim by default. A head_dim the kernels do not run is padded with zeros (padded_head_dim).rxNc3lK|].}tjj|dz fV/dS)rN)rnn functionalpad)rMtensorr paddeds r rNzattention..sBccv58&**6Av7H3IJJccccccrFr) memory_format.)rrr5rPr/r SCALED_V_SETTINGSSETTINGSr~rvrrr empty_likecontiguous_format) qkv_list recycle_qrattention_maskr]rr{r'ru quantizedr0r rs @@r attentionrs>GAq! NNwr{H&x::S[ScIhrsvv  000000  WWWWWWHHHHHH%%%%   PPPPPPAAAAAA      (jjjjjjjjj(((#''N[[[( u %  qq ! ! UUU ffffZZZ  4;;; 4AAA  4BBB MMM[[[ ^^^ 1111*111 \ \ \ \888ffff)-)-)-)-\]]]AAA (((eee\\\>>>>>>s$;;A AA