oMjSSKJrJr SSKrSSKJr SSKJs Jr SSK J r J r SSK J r JrJr SSKJr SSKJrJrJrJrJrJr SS KJrJrJr SS KJr SS KJ r J!r!J"r"J#r#J$r$ \"5(aSSK%r&OSr&\ RN"\(5r)"S S 5r*"SS5r+S\RXS\RZS\.S\.4Sjr/\"SS\RX55r0\"SS\RX55r1\"SS\RX55r2"SS\RX5r3\"SS\RX55r4"SS \RX5r5\"S!S"\RX55r6"S#S$\RX5r7g)%)AnyCallableN) deprecatelogging)is_torch_npu_availableis_torch_xla_availableis_xformers_available)maybe_allow_in_graph)GEGLUGELUApproximateGELUFP32SiLULinearActivationSwiGLU) AttentionAttentionProcessorJointAttnProcessor2_0)SinusoidalPositionalEmbedding) AdaLayerNormAdaLayerNormContinuousAdaLayerNormZeroRMSNormSD35AdaLayerNormZeroXc`\rSrSr\S\\\44Sj5rS\\\\4-4Sjr Sr Sr Sr g ) AttentionMixin'returnc^0nS[S[RRS[[[ 44U4SjjmUR 5Hup#T"X#U5 M U$)z Returns: `dict` of attention processors: A dictionary containing all attention processors used in the model with indexed by its weight name. namemodule processorsc>[US5(aUR5X S3'UR5Hup4T"USU3XB5 M U$)N get_processor .processor.)hasattrr%named_children)r!r"r#sub_namechildfn_recursive_add_processorss ڨC:\Users\johnn\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.13_qbz5n2kfra8p0\LocalCache\local-packages\Python313\site-packages\diffusers/models/attention.pyr,CAttentionMixin.attn_processors..fn_recursive_add_processors2sZv//282F2F2H V:./#)#8#8#:+tfAhZ,@%T$; )strtorchnnModuledictrr))selfr#r!r"r,s @r-attn_processorsAttentionMixin.attn_processors(sb  c 588?? X\]`bt]tXu !//1LD 'j A2r/ processorc d^[URR55n[U[5(a-[U5U:wa[ S[U5SUSUS35eS[ S[RR4U4SjjmUR5Hup4T"X4U5 M g) a Sets the attention processor to use to compute attention. Parameters: processor (`dict` of `AttentionProcessor` or only `AttentionProcessor`): The instantiated processor class or a dictionary of processor classes that will be set as the processor for **all** `Attention` layers. If `processor` is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors. z>A dict of processors was passed, but the number of processors z0 does not match the number of attention layers: z. Please make sure to pass z processor classes.r!r"c >[US5(aJ[U[5(dURU5 O#URUR US355 UR 5Hup4T"USU3XB5 M g)N set_processorr&r')r( isinstancer4r;popr))r!r"r8r*r+fn_recursive_attn_processors r-r>FAttentionMixin.set_attn_processor..fn_recursive_attn_processorUssv//!)T22((3(($z7J)KL#)#8#8#:+tfAhZ,@%S$;r/N) lenr6keysr<r4 ValueErrorr0r1r2r3r))r5r8countr!r"r>s @r-set_attn_processor!AttentionMixin.set_attn_processor@sD((--/0 i & &3y>U+BPQTU^Q_P`a005w6QRWQXXkm   Tc T588?? T!//1LD 'i @2r/cHURR5H3upS[URR5;dM*[ S5e UR 5H=n[U[5(dMUR(dM-UR5 M? g)z Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused. AddedzQ`fuse_qkv_projections()` is not supported for models having added KV projections.N) r6itemsr0 __class____name__rBmodulesr<AttentionModuleMixin_supports_qkv_fusionfuse_projections)r5_attn_processorr"s r-fuse_qkv_projections#AttentionMixin.fuse_qkv_projectionsbs| "&!5!5!;!;!= A#n66??@@ !tuu">llnF&"677F [!WARNING] > This API is 🧪 experimental. N)rKr<rLrMunfuse_projections)r5r"s r-unfuse_qkv_projections%AttentionMixin.unfuse_qkv_projectionsos9 llnF&"677F. A,>cK]F]A^,^ AD *,r/rc \rSrSrSr/rSrSrS\SS4Sjr S(S\ SS 4S jjr S \ 4S jr S \ SS4SjrS)S\ S\\ S-S4S-SS4SjjrS*S\ S\S-SS4Sjjr\R("5S5r\R("5S5rS\SS4SjrS\R2S\R24SjrS+S\R2S\S\R24SjjrS*S\R2S\R2S \R2S-S\R24S!jjrS+S \R2S"\S#\S\S\R24 S$jjrS%\R2S\R24S&jrS'rg),rLyNTFr8rcj[US5(a[UR[RR 5(ai[U[RR 5(d@[ RSURSU35 URRS5 Xlg)zu Set the attention processor to use. Args: processor (`AttnProcessor`): The attention processor to use. r8z-You are removing possibly trained weights of z with N) r(r<r8r1r2r3loggerinfo_modulesr=)r5r8s r-r;"AttentionModuleMixin.set_processorsx D+ & &4>>588??;;y%((//:: KKGGWW]^g]hi j MM  k *"r/return_deprecated_lorarc*U(d UR$g)z Get the attention processor in use. Args: return_deprecated_lora (`bool`, *optional*, defaults to `False`): Set to `True` to return the deprecated LoRA attention processor. Returns: "AttentionProcessor": The attention processor in use. N)r8)r5rds r-r%"AttentionModuleMixin.get_processors&>> !&r/backendcSSKJn URR5Vs1sHo3RiM nnX;a"[ SU<S3SR U5-5eU"UR55nXRl gs snf)Nr )AttentionBackendNamez `backend=z ` must be one of the following: z, ) attention_dispatchri __members__valuesvaluerBjoinlowerr8_attention_backend)r5rgrixavailable_backendss r-set_attention_backend*AttentionModuleMixin.set_attention_backendsw?? ""=1r/use_xla_flash_attentionpartition_spec.chU(a[5(d [S5eURS5 g)a Set whether to use XLA flash attention from `torch_xla` or not. Args: use_xla_flash_attention (`bool`): Whether to use pallas flash attention kernel from `torch_xla` or not. partition_spec (`tuple[]`, *optional*): Specify the partition specification if using SPMD. Otherwise None. is_flux (`bool`, *optional*, defaults to `False`): Whether the model is a Flux model. ztorch_xla is not available _native_xlaN)r rxrs)r5r{r|is_fluxs r-set_use_xla_flash_attention0AttentionModuleMixin.set_use_xla_flash_attentions*" #)++!">?? ""=1r/'use_memory_efficient_attention_xformers attention_opcU(a[5(d [SSS9e[RR 5(d [ S5e[5(aLSnUbUupEUR tp6[R"SSUS9n[RRXwU5nURS5 gg![anUeSnAff=f) ax Set whether to use memory efficient attention from `xformers` or not. Args: use_memory_efficient_attention_xformers (`bool`): Whether to use memory efficient attention from `xformers` or not. attention_op (`Callable`, *optional*): The attention operation to use. Defaults to `None` which uses the default attention operation from `xformers`. zeRefer to https://github.com/facebookresearch/xformers for more information on how to install xformersxformers)r!zvtorch.cuda.is_available() should be True but is False. xformers' memory efficient attention is only available for GPU N)r r(cudadevicedtype) r ModuleNotFoundErrorr1r is_availablerBSUPPORTED_DTYPESrandnxopsopsmemory_efficient_attention Exceptionrs) r5rrrop_fwop_bwrOqes r-+set_use_memory_efficient_attention_xformers@AttentionModuleMixin.set_use_memory_efficient_attention_xformerss 3(**){#ZZ,,.. / ,.. $'3+7LE(-(>(>IE!KK 6O HH??aH**:61 3*!GsAB>> CC  Cc UR(d-[RURRS35 g[ USS5(agUR RRRnUR RRRn[US5(GakUR(GaY[R"URRRUR RR/5nUR"SnUR"Sn[$R&"XEUR(XS9UlUR*RR-U5 [US 5(aUR((ay[R"URR.RUR R.R/5nUR*R.R-U5 GO[R"UR RRURRRUR RR/5nUR"SnUR"Sn[$R&"XEUR(XS9UlUR0RR-U5 [US 5(aUR((a[R"UR R.RURR.RUR R.R/5nUR0R.R-U5 [ US S5Gb[ US S5Gb[ US S5Gb[R"UR2RRUR4RRUR6RR/5nUR"SnUR"Sn[$R&"XEUR8XS9UlUR:RR-U5 UR8(a[R"UR2R.RUR4R.RUR6R.R/5nUR:R.R-U5 S Ulg)zU Fuse the query, key, and value projections into a single projection for efficiency. zK does not support fusing QKV projections, so `fuse_projections` will no-op.Nfused_projectionsFis_cross_attentionr r)biasrruse_bias add_q_proj add_k_proj add_v_projT)rMr`debugrIrJgetattrto_qweightdatarrr(rr1catto_kto_vshaper2Linearrto_kvcopy_rto_qkvrrradded_proj_bias to_added_qkvr)r5rrconcatenated_weights in_features out_featuresconcatenated_biass r-rN%AttentionModuleMixin.fuse_projectionss(( LL>>**++vw   4,e 4 4 !!&&--   %%++ 4- . .43J3J3J#(99dii.>.>.C.CTYYEUEUEZEZ-[#\ .44Q7K/55a8L;4==Y_mDJ JJ   # #$8 9tZ((T]]$)IItyy~~/B/BDIINNDWDW.X$Y! %%&78$)99dii.>.>.C.CTYYEUEUEZEZ\`\e\e\l\l\q\q-r#s .44Q7K/55a8L))KDMMZ`nDK KK   $ $%9 :tZ((T]]$)IItyy~~/B/BDIINNDWDWY]YbYbYgYgYlYl.m$n!   &&'89 D, - 9lD1=lD1=#(99'',,doo.D.D.I.I4??KaKaKfKfg$ /44Q7K/55a8L " 0D0DV!D     $ $ * *+? @##$)II__))..0D0D0I0I4??K_K_KdKde%!!!&&,,->?!%r/c UR(dg[USS5(dg[US5(a [US5 [US5(a [US5 [US5(a [US5 SUlg)zL Unfuse the query, key, and value projections back to separate projections. NrFrrr)rMrr(delattrr)r5s r-rT'AttentionModuleMixin.unfuse_projections:st(( t0%88  4 " " D( # 4 ! ! D' " 4 ( ( D. )!&r/ slice_sizec[US5(a.Ub+XR:a[SUSURS35eSnUbURS5nUcUR 5nUR U5 g)z Set the slice size for attention computation. Args: slice_size (`int`): The slice size for attention computation. sliceable_head_dimNz slice_size z has to be smaller or equal to r'sliced)r(rrB_get_compatible_processordefault_processor_clsr;)r5rr8s r-set_attention_slice(AttentionModuleMixin.set_attention_sliceTs 4- . .:3Ij[r[rNr{:,6UVZVmVmUnnopq q   !66x@I  224I 9%r/tensorcURnURup4nURX2-X$U5nURSSSS5RX2-XEU-5nU$)z Reshape the tensor from `[batch_size, seq_len, dim]` to `[batch_size // heads, seq_len, dim * heads]`. Args: tensor (`torch.Tensor`): The tensor to reshape. Returns: `torch.Tensor`: The reshaped tensor. rrr )headsrreshapepermute)r5r head_size batch_sizeseq_lendims r-batch_to_head_dim&AttentionModuleMixin.batch_to_head_dimks_JJ #)<< S 7SQ1a+33J4KW\eVef r/out_dimcURnURS:XaURupEnSnOURupGpVURXEU-X6U-5nUR SSSS5nUS:XaURXC-XW-Xc-5nU$)z Reshape the tensor for multi-head attention processing. Args: tensor (`torch.Tensor`): The tensor to reshape. out_dim (`int`, *optional*, defaults to `3`): The output dimension of the tensor. Returns: `torch.Tensor`: The reshaped tensor. rr rr)rndimrrr)r5rrrrrr extra_dims r-head_to_batch_dim&AttentionModuleMixin.head_to_batch_dim{sJJ ;;! '-|| $JI28,, /J7 i,?S\L\]1a+ a<^^J$:G