j2dZddlZddlmZddlZddlmZddlmZmZddl m Z m Z m Z ddl mZ ddl mZn #e$rdZYnwxYwgd Zd Zd Zd Zdad aeddZeddZdZdZdZdZejd*dZ d*dZ!d+dZ"dZ#dZ$e$Z%dZ&dZ'd,dZ(dZ)ej*fd Z+ej,Gd!d"Z-Gd#d$Z.d%Z/d&Z0d*d'Z1d(Z2ej d-d)Z3dS).aVRAM savings shared by model attentions: a model calls qkv_attention() (joint_qkv_attention() for several token streams) instead of projecting q/k/v and calling pay_attention. - SageAttention 2 (sm86/sm89/sm120, and its masked Triton path on every architecture) and SageAttention (1, without a mask): v, k and q are quantized as soon as each is projected (staged functions of shared.sage2_core / shared.sage1_core), so at most one of them exists in 16 bits (with the Triton kernels v stays in fp16); same results as pay_attention. - The three projections share one input prepared by mmgp (offload.prepare_linear_input) when their weight format supports it: the input is quantized once instead of once per projection, and the 16-bit input is released before the projections. - Attention Head Split (Performance tab, off by default): long sequences (all batch items together) are attended one group of heads at a time into one output. q/k/v rows, per-head norms, RoPE and attention are per head; norms over all heads get per token statistics computed beforehand. - Joint attentions: the projections of several streams (text and image tokens) are joined along the tokens in their order, and keys and values computed beforehand (a cached prefix) can be put before the projected ones. - Fused quantization (fused_quant): with a staged SageAttention 2 core, a model whose q/k norm and RoPE have a fused kernel (shared.kernels.ops.split_half_norm_rope_quant_) quantizes q and k straight from the projection output, without the in-place 16-bit pass and the core's separate quantizers (and, for k, its mean and subtraction passes). - Cached prefixes (cache_prefix, q8_prefix; PrefixCache for a model's layers, in VRAM or streamed from RAM): keys and values computed once and attended at every step (reference images, a history) kept quantized, about half of their 16-bit memory. With a staged SageAttention core they keep its own quantized layout and are attended in a pass of their own, merged with the projected keys by log-sum-exp: the error of attending the joined keys, and no joined 16-bit copy at each step. Otherwise they are q8 (int8 per token, head and 32 channels): dequantized into the joined keys and values. N) namedtuple)tqdm)offloadruntime) pay_attention sage_stagedsage_attention_mask)dispatch)kv_prefix_triton))Offr)zLow: saves some VRAM)z&Medium (good balance): saves more VRAM)zHigh: saves the most VRAM)r i iF HeadGroupzq kv mean_squaresKVPrefixz!k k_scale k_mean v v_scale v_mean i)@c$t|adSN)int head_split)levels r-BO T TF ~XRVWWWWND("*1+x!}hqsmZ8E NNt--hAA4CYCYZdfnCoCopI))%::IaL   8 < <zhead_groups..Vs3VV6x&?PTU?U?U?U?U?U?UVVr cBt|tz |fSr)abs_TARGET_GROUPS)r:rs rzhead_groups..VsClopvzHINzOqOmPmPRXlYr )key)rMIN_SPLIT_TOKENSminrange)r;tokensrs` `r head_groupsrFPs -JJUE zzV...q VVVVU1hl%;%;VVV]Y]Y]Y]Y Z Z ZZr c \| |jdn|}tdttd|ddddfz}t |||D]D}t ||z|}|dd||f}|||||~EdS)ztensor[:, first:last] = fn(tensor[:, first:last], first, last) by chunks of tokens of [start, stop), in place: fn computes each row on its own (norm, RoPE, possibly in place), so the result is that of fn on the whole range, with a bounded scratch.Nr )r&max _ROW_ELEMENTSnumelrDrCcopy_)tensorfnstartstopstepfirstlastparts rrows_rTYs#l6<??D q-3q&BQB-*=*=*?*?#@#@@ A ADudD))54<&&aaatm$ 22dE4(())) D r ctduo@|jdtvo,tj|t jt jfdS)zvThe Triton kernels of kv_prefix_triton for this 16-bit output: compute capability 7.5 and up (BF16 as bits below 8.0).Nr"))dtypescapability_floor) _kv_tritonr& _TRITON_DIMSr cuda_inferencetorchbfloat16float16)outs r_kernels_eligibleraesH d " nsy} 'D n'U^U]4Sflmmmor cTtj|jtj|j}tjg|jdd|jdt zRtj|j}td|jdtD] }|tz}|dd||f ddt f}| d dd}tj||dddd |dd||f<||dd||f<~~||fS) zx (batch, tokens, ..., channels) in the q8 format: (int8 data, FP16 scales (..., channels // 32)), each group of 32 channels scaled by max|x| / 127 (that of kv_formats.Q8).dtyper%Nr"rr g:0yE>i)r]emptyr&int8r%GROUPr_rD_Q8_ROWSfloat unflattenr>amaxdiv_ clamp_min_div unsqueezeround_clamp_flatten)xdatascalesrNrOrSscales r quantize_q8ryksw ;qwej B B BD [>173B3<>)=>>em\]\d e e eFq!'!*h//xE$J%%''11"r5kBB ##((--88>>#ieoob.A.ABBIIKKRRSWY\]]eefhiiQQQd ] %qqq%*} %% <r ct|rtj|||Std|jdt D]}|t z}|dd||fddtf|dd||f dz d|dd||f<|S)zqout (batch, tokens, heads, head_dim) 16-bit, written: q8 data (batch or 1, ...) with its scales, through strides.rr Nr"rf) rarZdequantize_q8_intorDr&rjrkrlrirqrt)r`rvrwrNrOs rr{r{}s @,S$???q#)A,11PPx"111eDj=17799CCBU TTW]^_^_^_afgkak^kWlWrWrWtWtW~W~@BXCXCCLLMOPPAAAuTzM Jr ct|rtj||||Std|jdt D]}|t z}|dddd||f|dddd||f}}t j||}t j||z t j||z }}||z} || z dd d|| z dd d}}|dd||f |z|dd||f |zz|dd||f<!|S)zIn place: out = the attention over both key sets of queries attended over one (out, lse: base-2 log-sum-exp (batch, heads, tokens)) and over the other (other, other_lse).rr Nrr") rarZ merge_lse_rDr&rjr]maximumexp2 transposerqrk) r`lseother other_lserNrOabtoptotals r _merge_lse_rsA$S#ui@@@q#)A,11__x111aaat#$i111eDj0@&A1mAq!!z!c'""EJq3w$7$71AE $$Q**44R881u9:O:OPQST:U:U:_:_`b:c:c1 E$J/5577!;eAAAuTzM>R>X>X>Z>Z]^>^^AAAuTzM Jr Tc|\}}||r t|j|jdnd}|t |ddr|d|d|j|jr|d}||d|jtj dd}| |}| |g||\}} | |r||g|\} } } nt|dc\} } } t!|| || | | St|\}} ~t|\} } t!|| d| | dS)aA cached prefix (KVPrefix) for joint_qkv_attention's kv_prefix, of keys and values (batch, tokens, kv_heads, head_dim) normalized and rotated, handed off in kv_list ([key, value], cleared; both are consumed). native: in the quantized layout of the staged SageAttention core that will attend it where that core supports it (unmasked attentions without attention_fn: pass False for the others), attended in a pass of its own and merged by log-sum-exp, the error of attending the joined keys; otherwise q8, dequantized into the joined keys and values at each attention (a small error on 16-bit backends). About half of the 16-bit memory either way. A block recorded for a CUDA graph keeps the settings of an eager one, so the cached steps can be recorded.r"Nrprefix_supportedr _sage_v_scalingF)r'rr%r&hasattrrrdprefix_settingsr shared_stateget prefix_meanprefix_quantize_k prefix_fp8_vr)_q8r) kv_listnativerAvaluer-r.r/meankk_scalevv_scalev_means r cache_prefixrsJC MMOOO7= G[SYr] 3 3 34F gfQi1CDDPQIcIcdjkldmoroy{~|EJFJFay''q 3:w?S?W?WXikp?q?qrr$$++SE8TBB 7   X & & 4!%!7!7!J!J Aw#&u::t LQ&7D!Wf===SJAw UJAw Awa$ 7 77r cTd||||fD\}}}}t||d||dS)zA cached prefix of keys and values already in the q8 format (shared.llm_engines.lmgp.layers.kv_formats.Q8: int8 (batch, tokens, kv_heads, head_dim), or without batch, with FP16 scales per 32 channels), as kept by an archive: not copied.c3tK|]3}|dkr|n|dV4dS)rrN)dimrqr9ts rr<zq8_prefix..s@ff15577a<.sh__!T6$EK %PTP[$\$\$\^bdjkk______r )tuplerrrr)prefixrds `rdequantize_prefixrs\ ____'-x&@68V^B\%]___ _ __r ceZdZUdZdZejdzed<dZejdzed<dZ e dzed<dZ e ed<dZ eed <d Zeed <dd Zd ZdZdZdZdS) PrefixLayerzZOne layer of a PrefixCache: its cached keys / values, 16-bit (k, v) or quantized (prefix).NrrrrindexFquantizeTrc|jrt||g|o|jnd}|jj}|zt ||jd|jr|n||fDd|jrtfd|D}ntt}}|jr ||_ dS||c|_ |_ dS)a>k / v (batch, tokens, kv_heads, head_dim), normalized and rotated (consumed when quantized); kit: attended through the unmasked attention of joint_qkv_attention (a native prefix then). With a RAM cache the layer goes to RAM, its tensors being those of its GPU slot (holding them for this pass already).)rNcg|]}||Srr8rs r z%PrefixLayer.store..s0s0s0sqefererererr T)currentc3>K|]}|dntVdSr)next)r9rslotss rr<z$PrefixLayer.store..s1#W#W1AIDD4;;#W#W#W#W#W#Wr ) rrrownerringiterstorerrrrrr)selfrrkitrrrs @rrzPrefixLayer.storesFJ]\q!fS-@T[AAAAX\z  DJ0s0st}=`VV[\^_Z`0s0s0s~BCCDDE} 0!#W#W#W#WPV#W#W#WXE{{DKK1 = " DKKKNDFDFFFr c&|jduo|jduSr)rrrs rrgzPrefixLayer.emptysv~5$+"55r c|rtd|jr|jn |j|jfS)z%The kv_prefix of joint_qkv_attention.$KV cache has not been populated yet.)rg RuntimeErrorrrrrrs r kv_prefixzPrefixLayer.kv_prefixs> ::<< GEFF F"mAt{{$&$&1AAr c|rtd|jrt|jn |j|jfS)zBThe 16-bit keys / values (attentions outside the kit: masks, NAG).r)rgrrrrrrrs rrzPrefixLayer.getsH ::<< GEFF F15T ---TVTVDTTr c0dx|_x|_|_dSr)rrrrs rr'zPrefixLayer.clears(,,,$+++r F)__name__ __module__ __qualname____doc__rr]Tensor__annotations__rrrrrrboolrrrgrrr'r8r rrrsdd!Au|d!!!!Au|d!!!"FHtO"""E3NNNHdFD"""""666BBB UUU -----r rc4eZdZdZd dZdZdZdZd d Zd S) PrefixCacheaCached keys / values of a model's attention layers: a prefix (reference images, a history) computed once and attended at the following passes. One PrefixLayer per block, which the model passes to its block: layer.store(k, v) where it computes them, layer.kv_prefix() for joint_qkv_attention afterwards. - quantize: INT8 (cache_prefix: about half the memory, the layout of SageAttention where it attends them). - ram: kept in RAM and streamed to the GPU block by block with the weights (mmgp.runtime.state_ring, locked RAM as far as the reserved RAM allows): the VRAM of two layers. blocks: the model's blocks (a ModuleList called one by one), whose forward then fetches the layer's cache; without ram, blocks may be a number of layers. clear() at the end of the generation (clear(keep_ring=True) to store the layers again, e.g. when step-dependent LoRAs change them).FTKV cachec|rtj||nd|_fdtt |t r|nt |D|_|jD] }||_ dS)Nc4g|]}t|S))rrr)r)r9rrrs rrz(PrefixCache.__init__..s;ZZZV[{&QQQZZZr ) r state_ringrrD isinstancerlenlayersr)rblocksrrramnamelayers `` r__init__zPrefixCache.__init__s8;EG&vt444 ZZZZZ_doy{ACFpGpGfXekekMPQWMXMX`Y`YZZZ [  EEKK  r c*t|jSr)rrrs r__len__zPrefixCache.__len__s4;r c|j|Sr)r)rrs r __getitem__zPrefixCache.__getitem__s{5!!r c@|jdSNr)rrgrs rrgzPrefixCache.emptys{1~##%%%r c|jD]}||j6|r|jn|jdSdSr)rr'rresetrelease)r keep_ringrs rr'zPrefixCache.clears^[  E KKMMMM 9 !* CDIOO    0A0A0C0C ! r N)FTFrr) rrrrrrrrgr'r8r rrrs{KK    """&&&DDDDDDr rc |jd}|d}||j|j||}|jjt jkr|||jjd|j |j z |jjdf| dr|jn t j } t| |jdddd|j |j f|jdddd|j |j f|| g|} ~ n(||j|j|j||} |t j|j|j|j||| g|d \} } ~~ ||||d \}} t+|| | | S) zThe staged attention of the quantized [q, k, v] written into out, merged with the queries' attention over the heads of a native cached prefix (two passes sharing the prefix's key mean).rr r"scale_v)rdNrcT) return_lse)r&prefix_krrrrdr]rh new_emptyrOrNrr_r{rr)prefix_vrr+rgr%r)r.r/r`r3rheadsbatch q_quantizedrrr prefix_out prefix_lsers r_attend_with_prefixr s IaLEA,K fhu==A x~## ufhnQ&7ek9QSYS[SabdSef|D|H|HIR|S|SofnqnwnwY^Yf gg5&(111aaaUZ1G+G"H&.YZYZYZ\]\]\]_d_jkpku_uYuJvwww  " "E7H 5 5 E MM&(FNFM5% P P!225;syPSPYbebl3m3m3mp{}~ABpCEMZ^2__J  1$$S)X$$OOHC sCZ 8 88r c|)|dks|j|dkr|n(|||j|j|jz S)z^mask restricted to the range heads along its head dimension dim, when it has one row per head.Nrr )rr&narrowrNrO)maskrrs r _head_slicer se<488::>>TZ_5I5I44t{{[^`e`kmrmwz{FnFPGPGGr c B|||f||fndz} t|| fg|||fi|S)aAttention of the q/k/v projections of x (handed off in x_list, shape (batch, tokens, features) or (tokens, features)), returned as (batch, tokens, heads, head_dim), batch 1 for a 2D x. See joint_qkv_attention for norm_rope and the options; extra_proj is a fourth per-head projection of x (heads x head_dim wide) computed by head groups from the same prepared input and handed to attention_fn as a third argument.Nr8joint_qkv_attention) x_listq_projk_projv_projrhead_dim norm_rope extra_projkwargs projectionss r qkv_attentionr%sF 66*z?Uzmm[]^K  56x ] ]V\ ] ]]r c >t||ddff|d||ffg|||fddi|S)zThe same projection/attention path with queries from x and keys/values from context, both handed off in lists. Head splitting considers the longer sequence, including a long video used as keys for short audio queries.N_crossTr) r context_listrrrrrrrs rcross_qkv_attentionr.sZ &$)= >PTV\^dOe@fg#Xy Q QAE QIO Q QQr c  3456789:;<=>?@AB|| dks| tdggc7=|D]x\}}|d}|7|dkr|n|d=|ydx}} r=dd=ddunt d=D} ot =ddk}7djdd 7D7djc39} r9dnt9@|p|}|r%t|3 rt9n@z|nd6| dnt7dj | |du}t|r|td tt 44ojdu;;r)||t%|dd std dgt 7z:t'=D]\}} rt)d |D}|r't)t*|}t/j|7|r'7|gdc}7|<t/j||:|<6dkrR3tdt59:Dzt6krt8st;jddad6iA3679:=Afd< 39<@fd}7:Afd}4;fdBd}|r6dkr| | |}n|r6dkrg}d|fd|ffD]\}}|6zd}!} t?6D]w5||t?5| z5dz| z ! d}"|!|"n|!"|"}!~"x||!#|zdt)|}|g|\}#}$;r?|#$|$j%j t.j&'dd}$|t|stQ||}|duo|duot| }%|%odup;}&|6z|6zc>8d}'t?6D]}tSt?|>z|dz>zt?|8z|dz8z|5|6dz k}(||d5j*|d5j+g??|r?d,n|d5j+|r|d5j*gnd})|(r || | ?d5|?d?d55?BfddD?dd<>8kr8>fd?ddD?dd<| )t[?t]|d5j*| d|}*n|) | ?5n | ?5|)}*6dkr|*cS|'|*/3@|f}'|'d5j*j05j*j1ddf2|*~*|r|d5j+nd}+|r|+,n|d5j+},| | |,5B|,d5j+g}-~,dd|#3|-|$g}.;r&i|$d|#4j5j+in|$}/6dkrF|s|d5j+}+|&r|d|+5|#|/nd}0|0V|d|+5B|+d5j+}+t|r |d|+5|#5|+g|/|.d<n |0d|.d<~+~0|d5j*}1||%r||1d5|#|$nd}0|0E||1d5t|r ||1d5n|}2|#6|1|$|.d<n||0dc}2|.d<~0;rto|#|$|1|.5j+cS|#8|1|.|$|2cS|d5j*}1|s|d5j+}+|(r ||&r||1|+5|#|/n|%r||1d5|#|$nd}0|0|0dZ||0|1nd|+5B|+d5j+}+t|r |d|+5|#5|+g|/|.d<n |0d|.d<~+|0Lt|r ||1d5nt]|d5j*}2|#6|1|$|.d<n"t]|d5j*|0dc}2|.d<~0|'|1/3@|f}';r;to|#|$|'d5j*j05j*j1ddf|.5j+n9|#8|'d5j*j05j*j1ddf|.|$|2~1|'S) a Attention of the q/k/v projections of several streams joined along the tokens in their order: streams is a list of (x_list, (q_proj, k_proj, v_proj)), each x handed off in its list, of shape (batch, tokens, features) or (tokens, features). Returns (batch, all tokens, heads, head_dim). norm_rope(query, key, group) normalizes and rotates in place query and/or key, the other one may be None: the joined projections, (batch, all tokens, heads of the group, head_dim), for the head ranges group.q and group.kv. With norm_spans_heads, a split group gets group.mean_squares to scale its rows; otherwise the tensors hold all heads. split_heads=False keeps all heads together. attention_fn: optional model-specific sparse attention accepting the disposable qkv_list and HeadGroup. It owns attention computation; projection preparation, head splitting and normalization still use this kit. Staged Sage quantization is disabled. kv_prefix: (key, value) of shape (batch, prefix tokens, kv_heads, head_dim), already normalized and rotated, put before the projected keys and values; or a cached prefix (cache_prefix, q8_prefix: KVPrefix), q8 dequantized into the joined keys and values, native attended in a pass of its own (a native prefix needs the staged core it was made for: no mask, no attention_fn), a prefix of one batch item serving all. attention_mask: pay_attention's mask, or on a staged core a callable mask(query, key, group) building the group's mask from its normalized and rotated query and key (the key with its prefix): called with the key alone first (it keeps what it needs, the key is then quantized and released), then with the query alone, returning the mask. kv_heads < heads: grouped query attention. norm_stats(project): optional model-native mean-square calculation, used only when splitting a norm spanning all heads; project(which, head_range) supplies disposable projections. Otherwise statistics are accumulated by groups below. norm_v(value, group): optional in-place value normalization, before prefix concatenation or quantization. When every K/V stream shares its k and v module, reuse its projection with independent tensors for normalization. softmax_scale: explicit SDPA scale; requires force_attention="sdpa". head_split_level: Attention Head Split level of this call instead of the Performance tab's (a model-specific sparse attention bounding its working set). A stream may carry a fourth projection: attention_fn then receives its head group's rows in a list. fused_quant(query, key, group, core, settings): optional norm + RoPE fused with a staged core's quantization (for example shared.kernels.ops.split_half_norm_rope_quant_), returning (q_quantized, k_quantized) for the tensors given (the other None) as the core's staged_quantize_q / staged_quantize_k do, or None to run norm_rope and the core's quantizers instead; used by the staged cores unless the attention_mask is a callable (it reads the normalized 16-bit q and k); with a kv_prefix only q is fused (the prefix joins the normalized 16-bit k), except with a native prefix (k fused with the prefix's mean: settings["k_mean"]).NsdpazMAn explicit softmax_scale requires force_attention='sdpa' and no attention_fnrrr c36K|]}|d|duVdS)r rNr8)r9moduless rr<z&joint_qkv_attention.._s4I~I~gn'RS*X_`aXbJbI~I~I~I~I~I~r rc(g|]}|jdS)r )r&)r9rus rrz'joint_qkv_attention..as0L0L0L0L0L0Lr zEA per-group attention mask (a callable) needs a staged attention corerzyA native cached prefix is attended by the unmasked staged attention it was made for (cache_prefix native=False otherwise)c3K|]}||V dSrr8)r9modules rr<z&joint_qkv_attention..os'OOvF.vs-!q!q-=VX`h`p&`p`p`p`p!q!qr z[Attention] Head Split is not applied: these q/k/v projections cannot be computed by groups of heads (weight format or LoRA type).Tc || |}}|Ctj|||jz|jz |dSdkr-|| |dS||f vr2|| |d ||f< ||fdddd|j|jfS)Nr"r )r linear_rowsrNrOview) rwhich head_ranger  linear_inputrr:rinputslengths linear_inputsrwholes rproject_streamz+joint_qkv_attention..project_stream}s1*51%8-:N   #&v|Z=MPX=XZdZiltZtuuzz|ACJKPCQSUW_`` ` Q;;6&-((--eWU^RRR R 5> & &"(&"7"7"<".projects  M!>uzz!!q%LL L w<<1  !>!UJ77 7 &w//  ME6!>% ;;D~(HABB(H(HII 111eEFN** + 1 1$ 7 7 7 VOE r cdSr)r')rrrsrrz$joint_qkv_attention..releases3  r c xr|Ss:tj|dz dddd|j|jf|gdS|dkrjjfn jjf\}}||j d|j d|j dzg|j ddR}t|ddd|j df|dddd|j|jf|dddd|j|jf|dd|j ddf ||S)Nr rrr) r]catrNrOrrrrrr&r{rK) rLrrrvrwrcachedrrs r with_prefixz(joint_qkv_attention..with_prefixs   M l9i 2111aaa9I*/9Y3YZ\bcijkkk k;@A:: Y%677IKYbYjKk f!!6<?DJqMFLQRO4S"gV\VbcdceceVf"g"ghh6!!!^djm^"34d111aaaAQR\RaAa;a6bdjklklklnononoq{rBCMCRrRlReS T T Tqqq$*Q-.. !''/// r )rfr"rr"rFc@g|]}||jSr8)kv)r9rgroupr,r!s rrz'joint_qkv_attention..s-^^^eKKII^^^r )r rcDg|]}|zdS)rr)repeat_interleave)r9rLkv_group_headsq_headss rrz'joint_qkv_attention..s3vvv_e 8 8N9RXY 8 Z Zvvvr )attention_maskr#r$ softmax_scale.k_mean)9 ValueErrorr'appendrrqallrr&rdsumrFrHrr%callablerrr+rrrdictfromkeysrlinear_rows_supportedprepare_linear_inputziprB_unsplit_noticerwriterDrksquare_add_rnrrrrr rqr#clonerrrrNrOrKr)prefix_mean_headsr(r*rr+)Cstreamsrrrr;norm_spans_heads split_headsrr)r# attention_fn norm_statsrnorm_vr*head_split_level fused_quantrstream_projectionsru shared_kvextrardr-rrx_holderrr mean_squaresrrrPsquaresrSr.r/fusing_qfusing_koutputrR extra_list attentionrArr2r3 k_settingsfusedr0rrr r$r:rr'rrrrrr(r,rErr!sC ` ` ` @@@@@@@@@@@@@@@@rrr5s B o&?&?^%?wvVWyZdGeGe%?UVV VFS[[(M#K00SSw  POOOOOOOG  4DMM'2233G  (&- @ @ S'-e}ot #HfUm#*#?'#R#RM%  zzec!q!qWmA\A\!q!q!qqqqvFFF # J\ ] ] ]"O EKKKKKKKKKKKK        L +FQJJ:+A!z'**  +fqjj Z!X7 N NLE5!VOT'Dv  wueEDL519:L&M&MNNTTVV^^``ddiqdrr")/$$w||D7I7ID    UX-= > > H H L L M M M M\**  h  ~++Hik6H'J^JbJbctv{J|J|}}H  %h~.F.F %0GGNT!^k&=^h~F^F^B^H9Y$.8&H#vox6/AG^ FvXX%%!)w1FGGuWeOehmpqhqvDhDJEJEGSTT " >57++WWQ-A-ABH OO9VHQK--///''!UXBVBV W W W27A''!UW--..TJ  !x{E*** Ihqk8A; 6 6 6^^^^^^W]^^^HQRRL.((vvvvviqrsrtrtiuvvv #)(;~_`bgbiCjCj}LX\lyzzz =G=OLL5999UaUabjlqs}U~U~ {{    ~",,eVUH-MNN 3 egl2AAA5 6 < >%u555Wbcqstv{v}W~W~D11%BBIaLL!,^Q!H!H%PQ( D)A,  >__eVUH%EFFF  i hsEGM%',r\s-*"""""" !!!!!!!!LLLLLLLLLL######=======JJJPPP   J{$7 8 8 :j"E F F     = = = =ZZZZ    ooo       88884888%*N____  *-*-*-*-*-*-*-*-ZDDDDDDDD@999(GGG ^^^^QQQAEQU;?ccccccs7AA